246 lines
9.3 KiB
Typst
246 lines
9.3 KiB
Typst
#let act(body) = [
|
||
#text(fill: rgb(230, 76, 0))[Action: #body]
|
||
]
|
||
|
||
#let dact(body) = [
|
||
#strike(stroke: 2pt+black)[#text(fill: rgb(230, 76, 0))[Action: #body]]
|
||
]
|
||
|
||
#align(center)[*Trust Management with Subjective Logic for Safety Critical Systems in Uncertain Environments*]
|
||
|
||
|
||
Review \#374A
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
|
||
Overall merit
|
||
|
||
*1.* Reject
|
||
|
||
|
||
Paper summary
|
||
|
||
The paper under review proposes a new system for trust management of
|
||
cyber physical system by using subjective logic and signal temporal
|
||
logic for trust assessment in uncertain environments. The theoretical
|
||
contribution of the paper is a development of a new cumulative fusion
|
||
formalism for subjective logic. This operator fuses multiple opinions
|
||
about the same proposition into a single, combined opinion, taking into
|
||
account the uncertainty inherent in each opinion. The authors prove that
|
||
this operator respects the goal relationship of the belief function.
|
||
Based on this new fusion formalism, they build a trust management
|
||
framework that aggregates data, generates a trust opinion over each time
|
||
interval end combines them with the ne fusion operator. The trust
|
||
management framework is then applied to two case studies.
|
||
|
||
|
||
*Strengths*
|
||
|
||
The new fusion operator is rigorously formalized and its correctness is
|
||
proven.
|
||
|
||
|
||
*Weaknesses*
|
||
|
||
- the use of STL for labeling the data is not explained, the used
|
||
formulas are very simple
|
||
#act[We could remove references to STL. The DSD does not truly use STL and the little it does is very simple and does not _require_ STL formalism. I agree it is cool to show that STL can be used but this is not the core of the proposed approach and it seems to confuse readers.]
|
||
- there is a large body of work on SLT based monitoring of CPS from
|
||
the formal methods community; it is unclear how this paper compares
|
||
and why these existing monitoring methods are not employed
|
||
#act[Again, maybe remove STL.]
|
||
- the overall motivation did not become clear and I do not see why I
|
||
need to use subjective logic on top of signal temporal logic
|
||
#act[Remove STL and reinforce the interest of subjective logic in the intro or presentation of the proposed approach]
|
||
- it is not obvious to me that combining opinions on trusts over
|
||
different time-windows is beneficial in CPS that are subject to
|
||
dynamic changes over time
|
||
#act[Provide a deeper analysis of the temporal relationship and the interest in not taking only the latest observation of trust.]
|
||
|
||
|
||
Review \#374B
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
|
||
Overall merit
|
||
|
||
*2.* Weak reject
|
||
|
||
|
||
Paper summary
|
||
|
||
This paper presents a novel trust management framework for
|
||
Cyber-Physical Systems (CPSs). Utilizing observations from target
|
||
systems, such as power consumption, the framework employs an activity
|
||
detector and an STL checker to classify these observations into
|
||
"satisfaction (+1)," "uncertainty (0)," or "violation (-1)" based on
|
||
predefined STL properties. For any fixed interval window, the framework
|
||
leverages subjective logic (SL) to compute a trust snapshot opinion,
|
||
which evaluates the system's trustworthiness within the current time
|
||
interval, as well as a trust index opinion, which provides an assessment
|
||
of the system's overall trustworthiness. Additionally, the framework
|
||
optionally include trust calibration actions, which can be manually
|
||
designed to bolster trust in the system if it falls below a predefined
|
||
threshold.
|
||
|
||
Meta review:
|
||
|
||
The reviewers thought the problem considered in this paper is important
|
||
and interesting, but it should be improved at least from the following
|
||
aspects: 1, the motivation is unclear, particular, why subjective logic
|
||
is necessary for dealing with trust, some real-world examples are
|
||
needed. 2, there is a huge bulk of work on monitoring of CPS, but
|
||
unfortunately, no comparison with them.
|
||
|
||
|
||
*Strengths*
|
||
|
||
1. Unlike existing works, the proposed framework accommodates the
|
||
possibility of "unknown" evidence, enabling it to handle uncertain
|
||
information effectively.
|
||
2. By leveraging subjective logic, the proposed framework offers a
|
||
quantitative assessment of the system's trustworthiness.
|
||
|
||
|
||
*Weaknesses*
|
||
|
||
1. The main technical contribution — extending beyond traditional
|
||
binary trust assessments to accommodate uncertainties — appears
|
||
limited. Incorporating 'unknown' into the domain of subjective logic
|
||
does not present significant challenges.
|
||
#act[Well *you* try to do it and *you* make a paper then!]
|
||
2. It is difficult to assess or reason why the obtained trust snapshot
|
||
opinion and trust index opinion accurately capture the
|
||
"trustworthiness" of the system.
|
||
|
||
|
||
Detailed comments
|
||
|
||
Based on the observed data series, the proposed framework calculates
|
||
values intended to capture the system's "trustworthiness." It is crucial
|
||
to demonstrate, either theoretically or experimentally, why the proposed
|
||
metrics effectively represent "trustworthiness," considering that
|
||
numerous alternative metrics could be proposed.
|
||
#act[Difficult to provide as there are no ground truth. Need to think about that.]
|
||
|
||
Minor:
|
||
|
||
On page 4, right column, the notation "W_2^F \oplus W_3^G" should be
|
||
corrected to "W_2^F \oplus W_2^G".
|
||
#act[fix that]
|
||
|
||
|
||
Review \#374C
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
|
||
Overall merit
|
||
|
||
*4.* Accept
|
||
|
||
|
||
Paper summary
|
||
|
||
The paper proposes a framework for utilizing Subjective Logic (SL) for
|
||
trust management. Core is the adoption of SL in two case studies (power
|
||
consumption of a NUC mini-PC from Intel, production data) for providing
|
||
a trust score even with uncertain or incomplete data. The framework is
|
||
intended for observations in cyber physical systems (CPS).
|
||
|
||
|
||
*Strengths*
|
||
|
||
- accountable derivation of adoption of SL for the case studies
|
||
- comprehensive description of both trust management framework and case studies
|
||
|
||
|
||
*Weaknesses*
|
||
|
||
- just a short discussion part
|
||
- a more comprehensive description of system reaction in the case of
|
||
violations and some evaluated evidence would support understanding
|
||
the effectiveness of the proposed framework for the use in CPS
|
||
#act[Not clear. Maybe talking about calibration actions (_reaction in the case of violation_)]
|
||
|
||
|
||
Detailed comments
|
||
|
||
the derivation of formulas given in Joesang "Subjective Logic" could be
|
||
shortened to more concise descriptions in favour of more detailed
|
||
description of interaction of the framework with the CPS in case of
|
||
violation
|
||
|
||
|
||
Review \#374D
|
||
|
||
------------------------------------------------------------------------
|
||
|
||
|
||
Overall merit
|
||
|
||
*1.* Reject
|
||
|
||
|
||
Paper summary
|
||
|
||
This paper proposes a model for quantifying trust in a system that uses
|
||
subjective logic. The model includes a trust index and a trust snapshot
|
||
for purposes of trust measurement, with trust calibration to allow the
|
||
user to adjust the trust. The design is evaluated with two case studies.
|
||
|
||
|
||
*Strengths*
|
||
|
||
Trust is an important concern in CPS, and it's interesting to see it
|
||
applies subjective logic for trust management purpose.
|
||
|
||
|
||
*Weaknesses*
|
||
|
||
- Poor motivation
|
||
#act[No. But we could still put more emphasis on the motivations.]
|
||
- Lack connection between the concepts and real-world scenarios
|
||
#act[Very no! Read the paper!]
|
||
- Unclear usefulness
|
||
#act[WHAT!? READ THE PAPER!]
|
||
- Lack comparisons to existing systems
|
||
#act[ok we could try to provide a baseline for comparison.]
|
||
|
||
|
||
Detailed comments
|
||
|
||
Managing trust in CPS is an important research problem, and the use of
|
||
subjective logics to this setting is interesting. Unfortunately, I feel
|
||
that the paper is rather immature and has a number of problems.
|
||
|
||
- The paper does not provide a convincing motivation to the proposed
|
||
framework. In the introduction, the description and importance of
|
||
trust management systems are so vague that it makes it impossible to
|
||
relate to real systems. A concrete example or use case should be
|
||
discussed to show where these systems are used and how bad
|
||
undertrust and overtrust would be.
|
||
|
||
- The concepts are not very well described. Where is the definition of
|
||
“integrity” (used in the problem statement)? Is it the same thing as
|
||
“trustworthiness”? How are “off”, “boot”, and “high load” in the
|
||
example related to integrity or trustworthiness? Real-world examples
|
||
should be provided to show what is a system with integrity and what
|
||
is a system without integrity.
|
||
#act[I still diasgree but for the few that read diagonaly we could maybe review the paragraphs explaining the real-world experiment and make it clearer.]
|
||
|
||
- The authors fail to address why the properties they prove in Sec.
|
||
IV-C are useful for real systems. What will be the consequences if
|
||
they don’t hold?
|
||
|
||
- There are existing intrusion detection systems using different
|
||
techniques. How is the proposed method superior to existing
|
||
solutions? The case study does not have any metrics to show how
|
||
good/accurate a method is.
|
||
#act[First, it does not need to be superior to be interesting. We are not selling vacume cleaner we are doing research. Second, we can look into providing a baseline with other methods.]
|
||
|
||
In short, while I find trust an interesting topic, the paper falls short
|
||
in several important aspects, making its benefits and contributions
|
||
questionable.
|