Distributed Aggregation Protocol for Privacy Preserving Measurement

1.2.1. Glossary of Terms

Aggregate result:

The output of the aggregation function. As defined in [VDAF].¶

Aggregate share:

A secret share of the aggregate result computed by each Aggregator and transmitted to the Collector. As defined in [VDAF].¶

Aggregation function:

The function computed over the measurements generated by the Clients and the aggregation parameter selected by the Collector. As defined in [VDAF].¶

Aggregation parameter:

Parameter selected by the Collector used to prepare a batch of measurements for aggregation. As defined in [VDAF].¶

Aggregator:

The party that receives report shares from Clients and validates and aggregates them with the help of the other Aggregator, producing aggregate shares for the Collector. As defined in [VDAF].¶

Batch:

A set of reports (i.e., measurements) that are aggregated into an aggregate result. As defined in [VDAF].¶

Batch bucket:

State associated with a given batch allowing the aggregators to perform incremental aggregation. Depending on the batch mode, there may be many batch buckets tracking the state of a single batch.¶

Batch interval:

A parameter of a query issued by the Collector that specifies the time range of the reports in the batch.¶

Client:

The party that generates a measurement and uploads a report, as defined in [VDAF]. Note the distinction between a DAP Client (distinguished in this document by the capital "C") and an HTTP client (distinguished in this document by the phrase "HTTP client"), as the DAP Client is not the only role that sometimes acts as an HTTP client.¶

Collector:

The party that selects the aggregation parameter and assembles the aggregate result from the aggregate shares constructed by the Aggregators. As defined in [VDAF].¶

Helper:

The Aggregator that executes the aggregation and collection interactions initiated by the Leader.¶

Input share:

An Aggregator's share of a measurement. The input shares are output by the VDAF sharding algorithm. As defined in [VDAF].¶

Output share:

An Aggregator's share of the refined measurement resulting from successful execution of VDAF preparation. Many output shares are combined into an aggregate share during VDAF aggregation. As defined in [VDAF].¶

Leader:

The Aggregator that coordinates aggregation and collection with the Helper.¶

Measurement:

A plaintext input emitted by a Client (e.g., a count, summand, or string), before any encryption or secret sharing is applied. Depending on the VDAF in use, multiple values may be grouped into a single measurement. As defined in [VDAF].¶

Minimum batch size:

The minimum number of reports that must be aggregated before a batch can be collected.¶

Public share:

An output of the VDAF sharding algorithm transmitted to each of the Aggregators. As defined in [VDAF].¶

Report:

A cryptographically protected measurement uploaded to the Leader by a Client. Includes a public share and a pair of report shares, one for each Aggregator.¶

Report share:

An input share encrypted under the HPKE public key of an Aggregator [HPKE]. The report share also includes some associated data used for processing the report.¶

Task:

A task in which measurements of an understood type will be reported by the Clients, aggregated by the Aggregators and received by the Collector. Many collections can be performed in the course of a single task.¶

2.4.1. Arity of Protocol Objects

Reports are 1 to 1 to with measurements. In this illustration, i distinct Clients upload a distinct report, but a single Client could upload multiple reports to a task (see Section 8.1 for some implications of this). The process of sharding measurements, constructing reports and uploading them is specified in Section 4.5.¶

Reports are many to 1 with aggregation jobs. The Leader assigns each of the i reports into one of j different aggregation jobs, which can be run in parallel by the Aggregators. Each aggregation job contains k reports. k is roughly i / j, but it is not necessary for aggregation jobs to have uniform size. See Section 6 for some discussion of performance implications for aggregation job scheduling strategies.¶

Report shares, input shares and output shares have a 1 to 1 to 1 relationship. Report shares are decrypted into input shares and then prepared into output shares. The scheduling of aggregation jobs and their execution by the Aggregators is specified in Section 4.6.¶

Output shares are accumulated into batch buckets, and so have a many to 1 relationship. In this example, we show one batch bucket for each aggregation job, but aggregation jobs and batch buckets are not necessarily 1 to 1. Multiple aggregation jobs could contribute to the same batch bucket, and a single aggregation job could contribute to multiple batch buckets. The assignation of output shares to batch buckets is specified in Section 4.6.3.3.¶

Using the Collector's query, each Aggregator will merge one or more batch buckets together into its aggregate share, meaning batch buckets are many to 1 with aggregate shares.¶

The Leader and Helper's aggregate shares are finally delivered to the Collector to be unsharded into the aggregate result. Since there are always exactly two Aggregators, aggregate shares are 2 to 1 with aggregate results. The collection interaction is specified in Section 4.7.¶

There can be many aggregate results for a single task. The Collection process may occur multiple times for each task, with the Collector obtaining multiple aggregate results. For example, imagine tasks where the Collector obtains aggregate results once an hour, or every time 10,000,000 reports are uploaded.¶

3.2.1. Variable-Length Vectors

Section 3.4 of [RFC8446] describes a syntax for variable-length vectors where a range of permitted lengths is provided. This document always specifies the lower length limit of variable-length vectors to be 0.¶

3.2.2. Structure Variants

Section 3.7 of [RFC8446] defines a syntax for structure fields whose values are constants. In this document, we do not use that notation, but use something similar to describe specific variants of structures that use an enumerated type as a discriminator, described in [RFC8446], Section 3.8.¶

For example, suppose we have an enumeration and a structure defined as follows:¶

enum {
  number(0),
  string(1),
  (255)
} ExampleEnum;

struct {
  uint32 always_present;
  ExampleEnum type;
  select (ExampleStruct.type) {
    case number: uint32 a_number;
    case string: opaque a_string<0..10>;
  };
} ExampleStruct;

Then we describe the specific variant of ExampleStruct where type == number with a variant block as follows:¶

variant {
  /* Field exists regardless of variant */
  uint32 always_present;
  ExampleEnum type = number;
  /* Only fields included in the `type == number`
     variant are described */
  uint32 a_number;
} ExampleStruct;

The protocol text accompanying this would explain how implementations should handle the always_present and a_number fields, but not the type field. This does not mean that the type field of ExampleStruct can only ever have value number; it means only that it has this type in this instance.¶

This notation can also be used in structures where the enum field is not used as a discriminant. For example, given the following structure containing an enumerator:¶

enum {
  something(0),
  something_else(1),
  (255)
} FailureReason;

struct {
  FailureReason failure_reason;
  opaque another_field<0..256>;
} FailedOperation;

The protocol text might include a description like:¶

variant {
  FailureReason failure_reason = something;
  opaque another_field<0..256>;
} FailedOperation;

4.1.1. Times, Durations and Intervals

uint64 Time;

Times are represented by POSIX timestamps, defined to be integers containing a number of seconds since the Epoch, defined in section 4.16 of [POSIX]. That is, the number of seconds after 1970-01-01 00:00:00 UTC, excluding leap seconds. One POSIX timestamp is said to be before (respectively, after) another POSIX timestamp if it is less than (respectively, greater than) the other value.¶

Every timestamp MUST be rounded down to the nearest multiple of the task's time_precision (Section 4.2). That is, the value is computed as:¶

rounded_timestamp = timestamp - (timestamp % time_precision)

A timestamp is malformed if received_timestamp % time_precision > 0.¶

uint64 Duration;

Durations of time are integers, representing a number of seconds not including leap seconds. They can be added to POSIX timestamps to produce other POSIX timestamps. The duration MUST be a multiple of time_precision. If duration % time_precision > 0, then the interval is malformed.¶

struct {
  Time start;
  Duration duration;
} Interval;

Intervals of time consist of a start time and a duration. Intervals are half-open; that is, start is included and (start + duration) is excluded. A time that is before the start of an Interval is said to be before that interval. A time that is equal to or after Interval.start + Interval.duration is said to be after the interval. A time that is either before or after an interval is said to be outside the interval. A time that is neither before nor after an interval is said to be inside or fall within the interval.¶

4.1.2. VDAF Types

The 16-byte ReportID is used as the nonce parameter for the VDAF shard and prep_init methods (see [VDAF], Section 5). Additionally, DAP includes messages defined in the VDAF specification encoded as opaque byte strings within various DAP messages. Thus, for a VDAF to be compatible with DAP, it MUST specify a NONCE_SIZE of 16 bytes, and MUST specify encodings for the following VDAF types:¶

• PublicShare¶

• InputShare¶

• AggParam¶

• AggShare¶

• PrepShare¶

• PrepMessage¶

4.2.1. Batch Modes, Batches, and Queries

An aggregate result is computed from a set of reports, called a "batch". The Collector requests the aggregate result by making a "query" and the Aggregators use this query to select a batch for aggregation.¶

The task's batch mode defines both how reports are assigned into batches, how these batches are addressed and the semantics of the query used for collection. Regardless of batch mode, each report can only ever be part of a single batch.¶

Section 5 defines the time-interval and leader-selected batch modes and discusses how new batch modes may be defined by future documents.¶

The query is issued to the Leader by the Collector during the collection interaction (Section 4.7). Information used to guide batch selection is conveyed from the Leader to the Helper when initializing aggregation jobs (Section 4.6) and finalizing the aggregate shares.¶

4.5.1. HPKE Configuration Request

Before the Client can upload its report to the Leader, it must know the HPKE configuration of each Aggregator. See Section 7 for information on HPKE algorithm choices.¶

Clients retrieve the HPKE configuration from each Aggregator by sending a GET to {aggregator}/hpke_config.¶

An Aggregator responds with an HpkeConfigList, with media type "application/dap-hpke-config-list". The HpkeConfigList contains one or more HpkeConfigs in decreasing order of preference. This allows an Aggregator to support multiple HPKE configurations and multiple sets of algorithms simultaneously.¶

HpkeConfig HpkeConfigList<0..2^16-1>;

struct {
  HpkeConfigId id;
  HpkeKemId kem_id;
  HpkeKdfId kdf_id;
  HpkeAeadId aead_id;
  HpkePublicKey public_key;
} HpkeConfig;

opaque HpkePublicKey<0..2^16-1>;
uint16 HpkeAeadId;
uint16 HpkeKemId;
uint16 HpkeKdfId;

The possible values for HpkeAeadId, HpkeKemId and HpkeKdfId are as defined in [HPKE], Section 7.¶

Aggregators MUST allocate distinct id values for each HpkeConfig in an HpkeConfigList.¶

The Client MUST abort if:¶

• the response is not a valid HpkeConfigList;¶

• the HpkeConfigList is empty; or¶

• no HPKE config advertised by the Aggregator specifies a supported KEM, KDF and AEAD algorithm triple.¶

Aggregators SHOULD use caching to permit client-side caching of this resource [RFC9111]. Aggregators can control cache lifetime with the Cache-Control header, using a value appropriate to the lifetime of their keys. Aggregators SHOULD favor long cache lifetimes to avoid frequent cache revalidation, e.g., on the order of days.¶

Aggregators SHOULD continue to accept reports with old keys for at least twice the cache lifetime in order to avoid rejecting reports.¶

GET /leader/hpke_config
Host: example.com

HTTP/1.1 200
Content-Type: application/dap-hpke-config-list
Cache-Control: max-age=86400

encoded([
  struct {
    id = 194,
    kem_id = 0x0010,
    kdf_id = 0x0001,
    aead_id = 0x0001,
    public_key = [0x01, 0x02, 0x03, 0x04, ...],
  } HpkeConfig,
  struct {
    id = 17,
    kem_id = 0x0020,
    kdf_id = 0x0001,
    aead_id = 0x0003,
    public_key = [0x04, 0x03, 0x02, 0x01, ...],
  } HpkeConfig,
])

4.5.2. Upload Request

Clients upload reports by sending a POST to {leader}/tasks/{task-id}/reports. The body is a Report, with media type "application/dap-report", structured as follows:¶

struct {
  ReportID report_id;
  Time time;
  Extension public_extensions<0..2^16-1>;
} ReportMetadata;

struct {
  ReportMetadata report_metadata;
  opaque public_share<0..2^32-1>;
  HpkeCiphertext leader_encrypted_input_share;
  HpkeCiphertext helper_encrypted_input_share;
} Report;

• report_metadata is public metadata describing the report.¶

  • report_id uniquely identifies the report. The Client MUST generate this by sampling 16 random bytes from a cryptographically secure random number generator.¶

  • time is the truncated (see Section 4.1.1) time at which the report was generated.¶

  • public_extensions is the list of public report extensions; see Section 4.5.3.¶

• public_share is the public share output by the VDAF sharding algorithm. The public share might be empty, depending on the VDAF.¶

• leader_encrypted_input_share is the Leader's encrypted input share.¶

• helper_encrypted_input_share is the Helper's encrypted input share.¶

Aggregators MAY require Clients to authenticate when uploading reports (see Section 8.3). If it is used, client authentication MUST use a scheme that meets the requirements in Section 3.3.¶

The handling of the upload request by the Leader MUST be idempotent as discussed in Section 9.2.2 of [RFC9110].¶

To generate a report, the Client begins by sharding its measurement into input shares and the public share using the VDAF's sharding algorithm (Section 5.1 of [VDAF]), using the report ID as the nonce:¶

(public_share, input_shares) = Vdaf.shard(
    "dap-15" || task_id,
    measurement,
    report_id,
    rand,
)

• task_id is the task ID.¶

• measurement is the plaintext measurement, represented as the VDAF's Measurement associated type.¶

• report_id is the corresponding value from ReportMetadata, used as the nonce.¶

• rand is a random byte string of length specified by the VDAF. Each report's rand MUST be independently sampled from a cryptographically secure random number generator.¶

The sharding algorithm will return two input shares. The first is the Leader's input share, and the second is the Helper's.¶

The Client then wraps each input share in the following structure:¶

struct {
  Extension private_extensions<0..2^16-1>;
  opaque payload<0..2^32-1>;
} PlaintextInputShare;

• private_extensions is the list of private report extensions for the given Aggregator (see Section 4.5.3).¶

• payload is the Aggregator's input share.¶

Next, the Client encrypts each PlaintextInputShare as follows:¶

enc, payload = SealBase(pk,
  "dap-15 input share" || 0x01 || server_role,
  input_share_aad, plaintext_input_share)

• pk is the public key from the Aggregator's HPKE configuration.¶

• 0x01 represents the Role of the sender (always the Client).¶

• server_role is the Role of the recipient (0x02 for the Leader and 0x03 for the Helper).¶

• plaintext_input_share is the Aggregator's PlaintextInputShare.¶

• input_share_aad is an encoded InputShareAad, constructed from the corresponding fields in the report per the definition below.¶

The SealBase() function is as specified in [HPKE], Section 6.1 for the ciphersuite indicated by the Aggregator's HPKE configuration.¶

struct {
  TaskID task_id;
  ReportMetadata report_metadata;
  opaque public_share<0..2^32-1>;
} InputShareAad;

If the upload request is malformed, the Leader aborts with error invalidMessage.¶

If the Leader does not recognize the task ID, then it aborts with error unrecognizedTask.¶

If the Leader does not recognize the config_id in the encrypted input share, it aborts with an error of type outdatedConfig. When the Client receives an outdatedConfig error, it SHOULD invalidate any cached HpkeConfigList and retry with a freshly generated Report. If this retried upload does not succeed, the Client SHOULD abort and discontinue retrying.¶

If a report's ID matches that of a previously uploaded report, the Leader MUST ignore it. In addition, it MAY alert the Client with error reportRejected.¶

The Leader MUST ignore any report pertaining to a batch that has already been collected (see Section 2.3 for details). The Leader MAY also abort with error reportRejected.¶

The Leader MUST ignore any report whose timestamp is outside of the task's time_interval. When it does so, it SHOULD also abort with error reportRejected.¶

The Leader may need to buffer reports while waiting to aggregate them (e.g., while waiting for an aggregation parameter from the Collector; see Section 4.7). The Leader SHOULD NOT accept reports whose timestamps are too far in the future. Implementors MAY provide for some small leeway, usually no more than a few minutes, to account for clock skew. If the Leader rejects a report for this reason, it SHOULD abort with error reportTooEarly. In this situation, the Client MAY re-upload the report later on.¶

If the report contains an unrecognized public report extension, or if the Leader's input share contains an unrecognized private report extension, then the Leader MUST ignore the report and MAY abort with error unsupportedExtension. If the Leader does abort for this reason, it SHOULD indicate the unsupported extensions in the resulting problem document via an extension member (Section 3.2 of [RFC9457]) unsupported_extensions on the problem document. This member MUST contain an array of numbers indicating the extension code points which were not recognized. For example, if the report upload contained two unsupported extensions with code points 23 and 42, the "unsupported_extensions" member would contain the JSON value [23, 42].¶

If the same extension type appears more than once among the public extensions and the private extensions in the Leader's input share, then the Leader MUST ignore the report and MAY abort with error invalidMessage.¶

Validation of anti-replay and extensions is not mandatory during the handling of upload requests to avoid blocking on storage transactions or decryption of input shares. The Leader also cannot validate the Helper's extensions because it cannot decrypt the Helper's input share. Validation of report IDs and extensions will occur before aggregation.¶

POST /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/reports
Host: example.com
Content-Type: application/dap-report

encoded(struct {
  report_metadata = struct {
    report_id = [0x0a, 0x0b, 0x0c, 0x0d, ...],
    time = 1741986088,
    public_extensions = [0x00, 0x00],
  } ReportMetadata,
  public_share = [0x0a, 0x0b, ...],
  leader_encrypted_input-share = struct {
    config_id = 1,
    enc = [0x0f, 0x0e, 0x0d, 0x0c, ...],
    payload = [0x0b, 0x0a, 0x09, 0x08, ...],
  } HpkeCiphertext,
  helper_encrypted_input-share = struct {
    config_id = 2,
    enc = [0x0c, 0x0d, 0x0e, 0x0f, ...],
    payload = [0x08, 0x00, 0x0a, 0x0b, ...],
  } HpkeCiphertext,
} Report)

HTTP/1.1 200

4.5.3. Report Extensions

Clients use report extensions to convey additional information to the Aggregators. Each ReportMetadata contains a list of extensions public to both aggregators, and each PlaintextInputShare contains a list of extensions private to the relevant Aggregator. For example, Clients may need to authenticate to the Helper by presenting a secret that must not be revealed to the Leader.¶

Each extension is a tag-length encoded value of the form:¶

struct {
  ExtensionType extension_type;
  opaque extension_data<0..2^16-1>;
} Extension;

enum {
  reserved(0),
  (65535)
} ExtensionType;

Field extension_type indicates the type of extension, and extension_data contains the opaque encoding of the extension.¶

Extensions are mandatory to implement. Unrecognized extensions are handled as specified in Section 4.6.2.4.¶

4.6.1. Eager Aggregation

In general, aggregation cannot begin until the Collector specifies a query and an aggregation parameter. However, depending on the VDAF and batch mode in use, it is often possible to begin aggregation as soon as reports arrive.¶

For example, Prio3 has just one valid aggregation parameter (the empty string), and so allows for eager aggregation. Both the time-interval and leader-selected batch modes defined in this document (Section 5) allow for eager aggregation, but future batch modes might preclude it.¶

Even when the VDAF uses a non-empty aggregation parameter, there still might be some applications in which the Aggregators can anticipate the parameter the Collector will choose and begin aggregation.¶

For example, when using Poplar1 (Section 8 of [VDAF]), the Collector and Aggregators might agree ahead of time on the set of candidate prefixes to use. In such cases, it is important that Aggregators ensure that the parameter eventually chosen by the Collector matches what they used. Depending on the VDAF, aggregating reports with multiple aggregation parameters may impact privacy. Aggregators must therefore ensure they only ever use the aggregation parameter chosen by the Collector.¶

4.6.2. Aggregate Initialization

Aggregation initialization accomplishes two tasks:¶

1. Determine which report shares are valid.¶

2. For each valid report share, initialize VDAF preparation (see Section 5.2 of [VDAF]).¶

The Leader and Helper initialization behavior is detailed below.¶

(state, outbound) = Vdaf.ping_pong_leader_init(
    vdaf_verify_key,
    "dap-15" || task_id,
    agg_param,
    report_id,
    public_share,
    plaintext_input_share.payload)

struct {
  ReportMetadata report_metadata;
  opaque public_share<0..2^32-1>;
  HpkeCiphertext encrypted_input_share;
} ReportShare;

struct {
  ReportShare report_share;
  opaque payload<0..2^32-1>;
} PrepareInit;

struct {
  BatchMode batch_mode;
  opaque config<0..2^16-1>;
} PartialBatchSelector;

struct {
  opaque agg_param<0..2^32-1>;
  PartialBatchSelector part_batch_selector;
  PrepareInit prepare_inits<0..2^32-1>;
} AggregationJobInitReq;

enum {
  continue(0),
  finished(1)
  reject(2),
  (255)
} PrepareRespState;

enum {
  reserved(0),
  batch_collected(1),
  report_replayed(2),
  report_dropped(3),
  hpke_unknown_config_id(4),
  hpke_decrypt_error(5),
  vdaf_prep_error(6),
  task_expired(7),
  invalid_message(8),
  report_too_early(9),
  task_not_started(10),
  (255)
} ReportError;

struct {
  ReportID report_id;
  PrepareRespState prepare_resp_state;
  select (PrepareResp.prepare_resp_state) {
    case continue: opaque payload<0..2^32-1>;
    case finished: Empty;
    case reject:   ReportError report_error;
  };
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = reject;
  ReportError report_error;
} PrepareResp;

(state, outbound) = Vdaf.ping_pong_helper_init(
    vdaf_verify_key,
    "dap-15" || task_id,
    agg_param,
    report_id,
    public_share,
    plaintext_input_share.payload)

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = reject;
  ReportError report_error = vdaf_prep_error;
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = continue;
  opaque payload<0..2^32-1> = outbound;
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = reject;
  ReportError report_error = commit_error;
} PrepareResp;

struct {
  PrepareResp prepare_resps<0..2^32-1>;
} AggregationJobResp;

plaintext_input_share = OpenBase(encrypted_input_share.enc, sk,
  "dap-15 input share" || 0x01 || server_role,
  input_share_aad, encrypted_input_share.payload)

PUT /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregation-job-init-req
Authorization: Bearer auth-token

encoded(struct {
  agg_param = [0x00, 0x01, 0x02, 0x04, ...],
  part_batch_selector = struct {
    batch_mode = BatchMode.leader_selected,
    config = encoded(struct {
      batch_id = [0x1f, 0x1e, ..., 0x00],
    } LeaderSelectedPartialBatchSelectorConfig),
  } PartialBatchSelector,
  prepare_inits,
} AggregationJobInitReq)

HTTP/1.1 200
Content-Type: application/dap-aggregation-job-resp

encoded(struct { prepare_resps } AggregationJobResp)

PUT /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregation-job-init-req
Authorization: Bearer auth-token

encoded(struct {
  agg_param = [0x00, 0x01, 0x02, 0x04, ...],
  part_batch_selector = struct {
    batch_mode = BatchMode.time_interval,
    config = encoded(Empty),
  },
  prepare_inits,
} AggregationJobInitReq)

HTTP/1.1 200
Location: /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=0
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=0
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Location: /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=0
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=0
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Content-Type: application/dap-aggregation-job-resp

encoded(struct { prepare_resps } AggregationJobResp)

4.6.3. Aggregate Continuation

In the continuation phase, the Leader drives the VDAF preparation of each report in the candidate set until the underlying VDAF moves into a terminal state, yielding an output share for each Aggregator or a rejection.¶

Continuation is only required for VDAFs that require more than one round. Single round VDAFs like Prio3 will never reach this phase.¶

struct {
  ReportID report_id;
  opaque payload<0..2^32-1>;
} PrepareContinue;

struct {
  uint16 step;
  PrepareContinue prepare_continues<0..2^32-1>;
} AggregationJobContinueReq;

(state, outbound) = Vdaf.ping_pong_helper_continued(
    "dap-15" || task_id,
    agg_param,
    state,
    inbound,
)

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = reject;
  ReportError report_error = vdaf_prep_error;
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = reject;
  ReportError report_error = commit_error;
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = continue;
  opaque payload<0..2^32-1> = outbound;
} PrepareResp;

variant {
  ReportID report_id;
  PrepareRespState prepare_resp_state = finished;
} PrepareResp;

POST /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregation-job-continue-req
Authorization: Bearer auth-token

encoded(struct {
  step = 1,
  prepare_continues,
} AggregationJobContinueReq)

HTTP/1.1 200
Content-Type: application/dap-aggregation-job-resp

encoded(struct { prepare_resps } AggregationJobResp)

POST /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregation-job-continue-req
Authorization: Bearer auth-token

encoded(struct {
  step = 1,
  prepare_continues,
} AggregationJobContinueReq)

HTTP/1.1 200
Location: /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=1
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=1
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Location: /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=1
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA?step=1
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Content-Type: application/dap-aggregation-job-resp

encoded(struct { prepare_resps } AggregationJobResp)

4.6.4. Aggregation Job Deletion

If for whatever reason, the Leader must abandon an aggregation job, it SHOULD let the Helper know it can clean up its state by sending a DELETE request to the job.¶

DELETE /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregation_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200

4.7.1. Collection Job Initialization

First, the Collector chooses a collection job ID, which MUST be unique within the scope of the corresponding DAP task.¶

To initiate the collection job, the Collector issues a PUT request to the collection job with media type "application/dap-collection-job-req", and a body structured as follows:¶

struct {
  BatchMode batch_mode;
  opaque config<0..2^16-1>;
} Query;

struct {
  Query query;
  opaque agg_param<0..2^32-1>; /* VDAF aggregation parameter */
} CollectionJobReq;

• query, the Collector's query. The content of this field depends on the indicated batch mode (Section 5).¶

• agg_param, an aggregation parameter for the VDAF being executed. This is the same value as in AggregationJobInitReq (see Section 4.6.2.1).¶

Collectors MUST authenticate their requests to Leaders using a scheme that meets the requirements in Section 3.3.¶

Depending on the VDAF scheme and how the Leader is configured, the Leader and Helper may already have prepared a sufficient number of reports satisfying the query and be ready to return the aggregate shares right away. However, this is not always the case. In fact, for some VDAFs, it is not be possible to begin running aggregation jobs (Section 4.6) until the Collector initiates a collection job. This is because, in general, the aggregation parameter is not known until this point. In certain situations it is possible to predict the aggregation parameter in advance. For example, for Prio3 the only valid aggregation parameter is the empty string.¶

The Leader MAY defer handling the collection request. In this case, it indicates that the collection job is not yet ready by immediately sending an empty response body. The Leader SHOULD include a Retry-After header field ([RFC9110], Section 10.2.3) to suggest a polling interval to the Collector. The Collector then polls the state of the job by sending GET requests to the collection job. The Collector SHOULD use each response's Retry-After header field to decide when to try again. The Leader responds the same way until either the job is ready, from which point it responds with a CollectionJobResp (defined below), or the job fails, from which point the Leader MUST abort with the error that caused the failure.¶

The Leader MAY instead handle the request immediately. It waits to respond to the Collector's PUT until the collection job is ready, in which case it responds with the CollectionJobResp, or the job fails, in which case the Leader MUST abort with the error that caused the failure.¶

If the job fails with invalidBatchSize, then the Collector MAY retry it later, once it believes enough new reports have been uploaded and aggregated to allow the collection job to succeed.¶

The Leader begins handling a CollectionJobReq by checking that it recognizes the task ID. If not, it MUST fail the collection job with error unrecognizedTask.¶

Next, the Leader checks that the indicated batch mode matches the task's batch mode. If not, it MUST fail the job with error invalidMessage.¶

Next, the Leader checks that the aggregation parameter is valid as described in Section 4.4. If not, it MUST fail the job with error invalidAggregationParameter.¶

The Leader then checks whether the Query in the Collector's request determines a batch that can be collected. If the query does not identify a valid set of batch buckets according to the criteria defined by the batch mode in use (Section 5), then the Leader MUST fail the job with error batchInvalid.¶

If any of the batch buckets identified by the query have already been collected, then the Leader MUST fail the job with error batchOverlap.¶

If aggregation is performed eagerly (Section 4.6.1), then the Leader checks that the aggregation parameter received in the CollectionJobReq matches the aggregation parameter used in each aggregation job pertaining to the batch. If not, the Leader MUST fail the job with error invalidMessage.¶

Having validated the CollectionJobReq, the Leader begins working with the Helper to aggregate the reports satisfying the query (or continues this process, depending on whether the Leader is aggregating eagerly; Section 4.6.1) as described in Section 4.6.¶

If the Leader has a pending aggregation job that overlaps with the batch for the collection job, the Leader MUST first complete the aggregation job before proceeding and requesting an aggregate share from the Helper. This avoids a race condition between aggregation and collection jobs that can yield batch mismatch errors.¶

If the number of validated reports in the batch is not equal to or greater than the task's minimum batch size, then the Leader SHOULD wait for more reports to be uploaded and aggregated and try the collection job again later. The Leader MAY give up on the collection job (for example, if it decides that no new reports satisfying the query are likely to ever arrive), in which case it MUST fail the job with error invalidBatchSize.¶

Once the Leader has validated the collection job and run to completion all the aggregation jobs that pertain to it, it obtains the Helper's aggregate share following the aggregate-share request flow described in Section 4.7.3. If obtaining the aggregate share fails, then the Leader MUST fail the collection job with the error that caused the failure.¶

Once the Leader has the Helper's aggregate share and has computed its own, the collection job is ready. Its results are represented by a CollectionJobResp, which is structured as follows:¶

struct {
  PartialBatchSelector part_batch_selector;
  uint64 report_count;
  Interval interval;
  HpkeCiphertext leader_encrypted_agg_share;
  HpkeCiphertext helper_encrypted_agg_share;
} CollectionJobResp;

A CollectionJobResp's media type is "application/dap-collection-job-resp". The structure includes the following:¶

• part_batch_selector: Information used to bind the aggregate result to the query. For leader-selected tasks, this includes the batch ID assigned to the batch by the Leader. The indicated batch mode MUST match the task's batch mode.¶

• report_count: The number of reports included in the batch.¶

• interval: The smallest interval of time that contains the timestamps of all reports included in the batch. In the case of a time-interval query (Section 5.1), this interval can be smaller than the one in the corresponding CollectionJobReq.query.¶

• leader_encrypted_agg_share: The Leader's aggregate share, encrypted to the Collector (see Section 4.7.6).¶

• helper_encrypted_agg_share: The Helper's aggregate share, encrypted to the Collector (see Section 4.7.6).¶

Once the Leader has constructed a CollectionJobResp for the Collector, the Leader considers the batch to be collected, and further aggregation jobs MUST NOT commit more reports to the batch (see Section 4.6.3.3).¶

Changing a collection job's parameters is illegal, so if there are further PUT requests to the collection job with a different CollectionJobReq, the Leader MUST abort with error invalidMessage.¶

PUT /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/collection_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-collection-job-req
Authorization: Bearer auth-token

encoded(struct {
  query = struct {
    batch_mode = BatchMode.leader_selected,
    query = encoded(Empty),
  } Query,
  agg_param = [0x00, 0x01, ...],
} CollectionJobReq)

HTTP/1.1 200
Content-Type: application/dap-collection

encoded(struct {
  part_batch_selector = struct {
    batch_mode = BatchMode.leader_selected,
    config = encoded(struct {
      batch_id = [0x1f, 0x1e, ..., 0x00],
    } LeaderSelectedPartialBatchSelectorConfig),
  } PartialBatchSelector,
  report_count = 1000,
  interval = struct {
    start = 1659544000,
    duration = 1000,
  } Interval,
  leader_encrypted_agg_share = struct { ... } HpkeCiphertext,
  helper_encrypted_agg_share = struct { ... } HpkeCiphertext,
} CollectionJobResp)

PUT /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/collection_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-collection-job-req
Authorization: Bearer auth-token

encoded(struct {
  query = struct {
    batch_mode = BatchMode.time_interval,
    query = encoded(struct {
      batch_interval = struct {
        start = 1659544000,
        duration = 10000,
      } Interval,
    } TimeIntervalQueryConfig),
  },
  agg_param = encoded(Empty),
} CollectionJobReq)

HTTP/1.1 200
Retry-After: 300

GET /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/collection_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Retry-After: 300

GET /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/collection_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Content-Type: application/dap-collection

encoded(struct {
  part_batch_selector = struct {
    batch_mode = BatchMode.time_interval,
    config = encoded(struct {
      interval = struct {
        start = 1659544000,
        duration = 10000,
      } Interval,
    } TimeIntervalBatchSelectorConfig)
  },
  report_count = 4000,
  interval = struct {
    start = 1659547000,
    duration = 1000,
  } Interval,
  leader_encrypted_agg_share = struct { ... } HpkeCiphertext,
  helper_encrypted_agg_share = struct { ... } HpkeCiphertext,
} CollectionJobResp)

4.7.2. Collection Job Deletion

The Collector can send a DELETE request to the collection job, which indicates to the Leader that it can abandon the collection job and discard all state related to it.¶

Collectors MUST authenticate their requests to Leaders using a scheme that meets the requirements in Section 3.3.¶

DELETE /leader/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/collection_jobs/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200

4.7.3. Obtaining Aggregate Shares

The Leader must compute its own aggregate share and obtain the Helper's encrypted aggregate share before it can complete a collection job.¶

First, the Leader retrieves all batch buckets (Section 4.6.3.3) associated with this collection job. The batch buckets to retrieve depend on the batch mode of this task:¶

• For time-interval (Section 5.1), this is all batch buckets whose batch bucket identifiers are contained within the batch interval specified in the CollectionJobReq's query.¶

• For leader-selected (Section 5.2), this is the batch bucket associated with the batch ID the Leader has chosen for this collection job.¶

The Leader then combines the values inside the batch bucket as follows:¶

• Aggregate shares are combined via Vdaf.merge(agg_param, agg_shares) (see [VDAF], Section 4.4), where agg_param is the aggregation parameter provided in the CollectionJobReq, and agg_shares are the (partial) aggregate shares in the batch buckets. The result is the Leader aggregate share for this collection job.¶

• Report counts are combined via summing.¶

• Checksums are combined via bitwise XOR.¶

A Helper aggregate share is identified by a 16-byte ID:¶

opaque AggregateShareID[16];

The Helper's aggregate share is an HTTP resource served by the Helper at the URL {helper}/tasks/{task-id}/aggregate_shares/{aggregate-share-id}. To obtain it, the Leader first chooses an aggregate share ID, which MUST be unique within the scope of the corresponding DAP task.¶

Then the Leader sends a PUT request to the aggregate share with the body:¶

struct {
  BatchMode batch_mode;
  opaque config<0..2^16-1>;
} BatchSelector;

struct {
  BatchSelector batch_selector;
  opaque agg_param<0..2^32-1>;
  uint64 report_count;
  opaque checksum[32];
} AggregateShareReq;

The media type of AggregateShareReq is "application/dap-aggregate-share-req". The structure contains the following parameters:¶

• batch_selector: The "batch selector", the contents of which depends on the indicated batch mode (see Section 5.¶

• agg_param: The encoded aggregation parameter for the VDAF being executed.¶

• report_count: The number number of reports included in the batch, as computed above.¶

• checksum: The batch checksum, as computed above.¶

Leaders MUST authenticate their requests to Helpers using a scheme that meets the requirements in Section 3.3.¶

The Helper MAY defer handling the aggregate share request. In this case, it indicates that the aggregate share is not yet ready by immediately sending an empty response body. The Helper SHOULD include a Retry-After header field ([RFC9110], Section 10.2.3) to suggest a polling interval to the Leader. The Leader then polls the state of the job by sending GET requests to the aggregate share. The Leader SHOULD use each response's Retry-After header field to decide when to try again. The Helper responds the same way until either the share is ready, from which point it responds with the AggregateShare (defined below), or the job fails, from which point it MUST abort with the error that caused the failure.¶

The Helper MAY instead handle the request immediately. It waits to respond to the Leader's PUT until the aggregate share is ready, in which case, it responds with the AggregateShare (defined below), or the job fails, in which case it MUST abort with the error that caused the failure.¶

The Helper first ensures that it recognizes the task ID. If not, it MUST fail the job with error unrecognizedTask.¶

The indicated batch mode MUST match the task's batch mode. If not, the Helper MUST fail the job with invalidMessage.¶

The Helper then verifies that the BatchSelector in the Leader's request determines a batch that can be collected. If the selector does not identify a valid set of batch buckets according to the criteria defined by the batch mode in use (Section 5), then the Helper MUST fail the job with error batchInvalid.¶

If any of the batch buckets identified by the selector have already been collected, then the Helper MUST fail the job with error batchOverlap.¶

If the number of validated reports in the batch is not equal to or greater than the task's minimum batch size, then the Helper MUST abort with invalidBatchSize.¶

The aggregation parameter MUST match the aggregation parameter used in aggregation jobs pertaining to this batch. If not, the Helper MUST fail the job with error invalidMessage.¶

Next, the Helper retrieves and combines the batch buckets associated with the request using the same process used by the Leader (described at the beginning of this section), arriving at its aggregate share, report count, and checksum values. If the Helper's computed report count and checksum values do not match the values provided in the AggregateShareReq, it MUST fail the job with error batchMismatch.¶

The Helper then encrypts agg_share under the Collector's HPKE public key as described in Section 4.7.6, yielding encrypted_agg_share. Encryption prevents the Leader from learning the actual result, as it only has its own aggregate share and cannot compute the Helper's.¶

Once the Helper has encrypted its aggregate share, the aggregate share job is ready. Its results are represented by an AggregateShare, with media type "application/dap-aggregate-share":¶

struct {
  HpkeCiphertext encrypted_aggregate_share;
} AggregateShare;

encrypted_aggregate_share.config_id is set to the Collector's HPKE config ID. encrypted_aggregate_share.enc is set to the encapsulated HPKE context enc computed above and encrypted_aggregate_share.ciphertext is the ciphertext encrypted_agg_share computed above.¶

After receiving the Helper's response, the Leader includes the HpkeCiphertext in its response to the Collector (see Section 4.7.5).¶

Once an AggregateShareReq has been constructed for the batch determined by a given query, the Helper considers the batch to be collected. The Helper MUST NOT commit any more output shares to the batch. It is an error for the Leader to issue any more aggregation jobs for additional reports that satisfy the query. These reports MUST be rejected by the Helper as described in Section 4.6.3.3.¶

Changing an aggregate share's parameters is illegal, so if there are further PUT requests to the aggregate share with a different AggregateShareReq, the Helper MUST abort with error invalidMessage.¶

Before completing the collection job, the Leader encrypts its aggregate share under the Collector's HPKE public key as described in Section 4.7.6.¶

PUT /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregate_shares/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregate-share-req
Authorization: Bearer auth-token

encoded(struct {
  batch_selector = struct {
    batch_mode = BatchMode.time_interval,
    config = encoded(struct {
      batch_interval = struct {
        start = 1659544000,
        duration = 10000,
      } Interval,
    } TimeIntervalBatchSelectorConfig),
  } BatchSelector,
  agg_param = [0x00, 0x01, ...],
  report_count = 1000,
  checksum = [0x0a, 0x0b, ..., 0x0f],
} AggregateShareReq)

HTTP/1.1 200
Content-Type: application/dap-aggregate-share

encoded(struct {
  encrypted_aggregate_share = struct { ... } HpkeCiphertext,
} AggregateShare)

PUT /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregate_shares/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Content-Type: application/dap-aggregate-share-req
Authorization: Bearer auth-token

encoded(struct {
  batch_selector = struct {
    batch_mode = BatchMode.time_interval,
    config = encoded(struct {
      batch_interval = struct {
        start = 1659544000,
        duration = 10000,
      } Interval,
    } TimeIntervalBatchSelectorConfig),
  } BatchSelector,
  agg_param = [0x00, 0x01, ...],
  report_count = 1000,
  checksum = [0x0a, 0x0b, ..., 0x0f],
} AggregateShareReq)

HTTP/1.1 200
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregate_shares/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Retry-After: 300

GET /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregate_shares/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200
Content-Type: application/dap-aggregate-share

encoded(struct {
  encrypted_aggregate_share = struct { ... } HpkeCiphertext,
} AggregateShare)

4.7.4. Aggregate Share Deletion

The Leader can send a DELETE request to the aggregate share, which indicates to the Helper that it can abandon the aggregate share and discard all state related to it.¶

Leaders MUST authenticate their requests to Helpers using a scheme that meets the requirements in Section 3.3.¶

DELETE /helper/tasks/8BY0RzZMzxvA46_8ymhzycOB9krN-QIGYvg_RsByGec/aggregate_shares/lc7aUeGpdSNosNlh-UZhKA
Host: example.com
Authorization: Bearer auth-token

HTTP/1.1 200

4.7.5. Collection Job Finalization

Once the Collector has received a collection job from the Leader, it can decrypt the aggregate shares and produce an aggregate result. The Collector decrypts each aggregate share as described in Section 4.7.6. Once the Collector successfully decrypts all aggregate shares, it unshards the aggregate shares into an aggregate result using the VDAF's unshard algorithm.¶

Let leader_agg_share denote the Leader's aggregate share, helper_agg_share denote the Helper's aggregate share, report_count denote the report count sent by the Leader, and agg_param denote the opaque aggregation parameter. The final aggregate result is computed as follows:¶

agg_result = Vdaf.unshard(agg_param,
                          [leader_agg_share, helper_agg_share],
                          report_count)

4.7.6. Aggregate Share Encryption

Encrypting an aggregate share agg_share for a given AggregateShareReq is done as follows:¶

(enc, payload) = SealBase(
    pk,
    "dap-15 aggregate share" || server_role || 0x00,
    agg_share_aad,
    agg_share)

• pk is the Collector's HPKE public key¶

• server_role is the Role of the encrypting server (0x02 for the Leader and 0x03 for a Helper)¶

• 0x00 represents the Role of the recipient (always the Collector)¶

• agg_share_aad is an AggregateShareAad (defined below).¶

The SealBase() function is as specified in [HPKE], Section 6.1 for the ciphersuite indicated by the HPKE configuration.¶

struct {
  TaskID task_id;
  opaque agg_param<0..2^32-1>;
  BatchSelector batch_selector;
} AggregateShareAad;

• task_id is the ID of the task the aggregate share was computed in.¶

• agg_param is the aggregation parameter used to compute the aggregate share.¶

• batch_selector is the is the batch selector from the AggregateShareReq (for the Helper) or the batch selector computed from the Collector's query (for the Leader).¶

The Collector decrypts these aggregate shares using the opposite process. Specifically, given an encrypted input share, denoted enc_share, for a given batch selector, decryption works as follows:¶

agg_share = OpenBase(
    enc_share.enc,
    sk,
    "dap-15 aggregate share" || server_role || 0x00,
    agg_share_aad,
    enc_share.payload)

• sk is the HPKE secret key¶

• server_role is the Role of the server that sent the aggregate share (0x02 for the Leader and 0x03 for the Helper)¶

• 0x00 represents the Role of the recipient (always the Collector)¶

• agg_share_aad is an AggregateShareAad message constructed from the task ID and the aggregation parameter in the collect request, and a batch selector. The value of the batch selector used in agg_share_aad is determined by the batch mode:¶

  • For time-interval (Section 5.1), the batch selector is the batch interval specified in the query.¶

  • For leader-selected (Section 5.2), the batch selector is the batch ID sent in the response.¶

The OpenBase() function is as specified in [HPKE], Section 6.1 for the ciphersuite indicated by the HPKE configuration.¶

5.1.1. Query Configuration

The payload of Query.config is¶

struct {
  Interval batch_interval;
} TimeIntervalQueryConfig;

where batch_interval is the batch interval requested by the Collector. The interval MUST be well-formed as specified in Section 4.1.1. Otherwise, the query does not specify a set of valid batch buckets.¶

5.1.2. Partial Batch Selector Configuration

The payload of PartialBatchSelector.config is empty.¶

5.1.3. Batch Selector Configuration

The payload of BatchSelector.config is¶

struct {
  Interval batch_interval;
} TimeIntervalBatchSelectorConfig;

where batch_interval is the batch interval requested by the Collector.¶

5.1.4. Batch Buckets

Each batch bucket is identified by an Interval whose duration is equal to the task's time_precision. The identifier associated with a given report is the unique such interval containing the timestamp of the report. For example, if the task's time_precision is 1000 seconds and the report's timestamp is 1729629081, the relevant batch bucket identifier is¶

struct {
  start = 1729629000,
  duration = 1000,
} Interval

The Query received by the Leader or BatchSelector received by the Helper determines a valid set of batch bucket identifiers if the batch interval's duration is greater than or equal to the task's time_precision.¶

A batch consists of a sequence of contiguous batch buckets. That is, the set of batch bucket identifiers for the batch interval is¶

[
  struct {
    start = batch_interval.start,
    duration = time_precision,
  } Interval,
  struct {
    start = batch_interval.start + time_precision,
    duration = time_precision,
  } Interval,
  ...
  struct {
    start = batch_interval.start + batch_interval.duration - time_precision,
    duration = time_precision,
  } Interval,
]

5.2.1. Query Configuration

They payload of Query.config is empty. The request merely indicates the Collector would like the next batch selected by the Leader.¶

5.2.2. Partial Batch Selector Configuration

The payload of PartialBatchSelector.config is:¶

struct {
  BatchID batch_id;
} LeaderSelectedPartialBatchSelectorConfig;

where batch_id is the batch ID selected by the Leader.¶

5.2.3. Batch Selector Configuration

The payload of BatchSelector.config is:¶

struct {
  BatchID batch_id;
} LeaderSelectedBatchSelectorConfig;

where batch_id is the batch ID selected by the Leader.¶

5.2.4. Batch Buckets

Each batch consists of a single bucket and is identified by the batch ID. A report is assigned to the batch indicated by the PartialBatchSelector during aggregation.¶

6.1.1. Client Capabilities

Clients have limited capabilities and requirements. Their only inputs to the protocol are (1) the parameters configured out of band and (2) a measurement. Clients are not expected to store any state across any upload flows, nor are they required to implement any sort of report upload retry mechanism. By design, the protocol in this document is robust against individual Client upload failures since the protocol output is an aggregate over all inputs.¶

6.1.2. Aggregator Capabilities

Leaders and Helpers have different operational requirements. The design in this document assumes an operationally competent Leader, i.e., one that has no storage or computation limitations or constraints, but only a modestly provisioned Helper, i.e., one that has computation, bandwidth, and storage constraints. By design, Leaders must be at least as capable as Helpers, where Helpers are generally required to:¶

• Support the aggregate interaction, which includes validating and aggregating reports; and¶

• Publish and manage an HPKE configuration that can be used for the upload interaction.¶

• Implement some form of batch-to-report index, as well as inter- and intra-batch replay mitigation storage, which includes some way of tracking batch report size. Some of this state may be used for replay attack mitigation. The replay mitigation strategy is described in Section 4.6.2.4.¶

Beyond the minimal capabilities required of Helpers, Leaders are generally required to:¶

• Support the upload interaction and store reports; and¶

• Track batch report size during each collect flow and request encrypted output shares from Helpers.¶

• Implement and store state for the form of inter- and intra-batch replay mitigation in Figure 3. This requires storing the report IDs of all reports processed for a given task. Implementations may find it helpful to track additional information, like the timestamp, so that the storage used for anti-replay can be sharded efficiently.¶

6.1.3. Collector Capabilities

Collectors statefully interact with Aggregators to produce an aggregate output. Their input to the protocol is the task parameters, configured out of band, which include the corresponding batch window and size. For each collect invocation, Collectors are required to keep state from the start of the protocol to the end as needed to produce the final aggregate output.¶

Collectors must also maintain state for the lifetime of each task, which includes key material associated with the HPKE key configuration.¶

6.4.1. Reducing Storage Requirements

In general, the Aggregators are required to keep state for tasks and all valid reports for as long as collection requests can be made for them. However, it is not necessary to store the complete reports. Each Aggregator only needs to store an aggregate share for each possible batch bucket i.e., the batch interval for time-interval or batch ID for leader-selected, along with a flag indicating whether the aggregate share has been collected. This is due to the requirement for queries to respect bucket boundaries. See Section 5.¶

However, Aggregators are also required to implement several per-report checks that require retaining a number of data artifacts. For example, to detect replay attacks, it is necessary for each Aggregator to retain the set of report IDs of reports that have been aggregated for the task so far. Depending on the task lifetime and report upload rate, this can result in high storage costs. To alleviate this burden, DAP allows Aggregators to drop this state as needed, so long as reports are dropped properly as described in Section 4.6.2.4. Aggregators SHOULD take steps to mitigate the risk of dropping reports (e.g., by evicting the oldest data first).¶

Furthermore, the Aggregators must store data related to a task as long as the current time is not after this task's task_interval. Aggregators MAY delete the task and all data pertaining to this task after the task_interval. Implementors SHOULD provide for some leeway so the Collector can collect the batch after some delay.¶

6.4.2. Distributed-systems and Synchronization Concerns

Various parts of a DAP implementation will need to synchronize in order to ensure correctness during concurrent operation. This section describes the relevant concerns and makes suggestions as to potential implementation tradeoffs.¶

• The upload interaction requires the Leader to ignore uploaded reports with a duplicated ID, including concurrently-uploaded reports. This might be implemented by synchronization or via an eventually-consistent process. If the Leader wishes to alert the Client with a reportRejected error, synchronization will be necessary to ensure all but one concurrent request receive the error.¶

• The Leader is responsible for generating aggregation jobs, and will generally want to place each report in exactly one aggregation job. (The only event in which a Leader will want to place a report in multiple aggregation jobs is if the Helper rejects the report with report_too_early, in which case the Leader can place the report into a later aggregation job.) This may require synchronization between different components of the system which are generating aggregation jobs. Note that placing a report into more than one aggregation job will result in a loss of throughput, rather than a loss of correctness, privacy, or robustness, so it is acceptable for implementations to use an eventually-consistent scheme which may rarely place a report into multiple aggregation jobs.¶

• Aggregation is implemented as a sequence of aggregation steps by both the Leader and the Helper. The Leader must ensure that each aggregation job is only processed once concurrently, which may require synchronization between the components responsible for performing aggregation. The Helper must ensure that concurrent requests against the same aggregation job are handled appropriately, which requires synchronization between the components handling aggregation requests.¶

• Aggregation requires checking and updating used-report storage as part of implementing replay protection. This must be done while processing the aggregation job, though which steps the checks are performed at is up to the implementation. The checks and storage require synchronization, so that if two aggregation jobs containing the same report are processed, at most one instance of the report will be aggregated. However, the interaction with the used-report storage does not necessarily have to be synchronized with the processing and storage for the remainder of the aggregation process. For example, used-report storage could be implemented in a separate datastore than is used for the remainder of data storage, without any transactionality between updates to the two datastores.¶

• The aggregation and collection interactions require synchronization to avoid modifying the aggregate of a batch after it has already been collected. Any reports being aggregated which pertain to a batch which has already been collected must fail with a batch_collected error; correctly determining this requires synchronizing aggregation with the completion of collection jobs (for the Leader) or aggregate share requests (for the Helper). Also, the Leader must complete all outstanding aggregation jobs for a batch before requesting aggregate shares from the Helper, again requiring synchronization between the Leader's collection and aggregation interactions. Further, the Helper must determine the aggregated report count and checksum of aggregated report IDs before responding to an aggregate share request, requiring synchronization between the Helper's collection and aggregation interactions.¶

8.6.1. Predictable or Enumerable Task IDs

This specification imposes no requirements on task IDs except that they be globally unique. One way to achieve this is to use random task IDs, but deployments can also use schemes like [I-D.draft-ietf-ppm-dap-taskprov-01] where task IDs are deterministically generated from some set of task parameters.¶

In such settings, deployments should consider whether an Aggregator acknowledging the existence of a task (by accepting report uploads or aggregation jobs, for example) could unintentionally leak information such as a label describing the task, the identities of participating Aggregators or the fact that some measurement is being taken at all.¶

Such enumeration attacks can be mitigated by incorporating unpredictable values into the task ID derivation. They do not, however, affect the protocol goals of privacy or robustness.¶

8.6.2. VDAF Verification Key Requirements

Knowledge of the verification key would allow a Client to forge a report with invalid values that will nevertheless pass verification. Therefore, the verification key must be kept secret from Clients.¶

Furthermore, for a given report, it may be possible to craft a verification key which leaks information about that report's measurement during VDAF preparation. Therefore, the verification key for a task SHOULD be chosen before any reports are generated. Moreover, it SHOULD be fixed for the lifetime of the task and not be rotated. One way to ensure that the verification key is generated independently from any given report is to derive the key based on the task ID and some previously agreed upon secret (verify_key_seed) between Aggregators, as follows:¶

verify_key = HKDF-Expand(
    HKDF-Extract(
        "verify_key",    # salt
        verify_key_seed, # IKM
    ),
    task_id,             # info
    VERIFY_KEY_SIZE,     # L
)

Here, VERIFY_KEY_SIZE is the length of the verification key, and HKDF-Extract and HKDF-Expand are as defined in [RFC5869].¶

This requirement comes from current security analysis for existing VDAFs. In particular, the security proofs for Prio3 require that the verification key is chosen independently of the generated reports.¶

8.6.3. Batch Parameters

An important parameter of a DAP deployment is the minimum batch size. If a batch includes too few reports, then the aggregate result can reveal information about individual measurements. Aggregators enforce the agreed-upon minimum batch size during collection, but implementations SHOULD also opt out of participating in a DAP task if the minimum batch size is too small. This document does not specify how to choose an appropriate minimum batch size, but an appropriate value may be determined from the differential privacy (Section 8.5) parameters in use, if any.¶

8.6.4. Task Configuration Agreement and Consistency

In order to execute a DAP task, it is necessary for all parties to ensure they agree on the configuration of the task. However, it is possible for a party to participate in the execution of DAP without knowing all of the task's parameters. For example, a Client can upload a report (Section 4.5) without knowing the minimum batch size that is enforced by the Aggregators during collection (Section 4.7).¶

Depending on the deployment model, agreement can require that task parameters are visible to all parties such that each party can choose whether to participate based on the value of any parameter. This includes the parameters enumerated in Section 4.2 and any additional parameters implied by report extensions Section 4.5.3 used by the task. Since meaningful privacy requires that multiple Clients contribute to a task, they should also share a consistent view of the task configuration.¶

9.1.1. "application/dap-hpke-config-list" media type

Type name:

application¶

Subtype name:

dap-hpke-config-list¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.5 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.2. "application/dap-report" media type

Type name:

application¶

Subtype name:

dap-report¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.5 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.3. "application/dap-aggregation-job-init-req" media type

Type name:

application¶

Subtype name:

dap-aggregation-job-init-req¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.6 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.4. "application/dap-aggregation-job-resp" media type

Type name:

application¶

Subtype name:

dap-aggregation-job-resp¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.6 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.5. "application/dap-aggregation-job-continue-req" media type

Type name:

application¶

Subtype name:

dap-aggregation-job-continue-req¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.6 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.6. "application/dap-aggregate-share-req" media type

Type name:

application¶

Subtype name:

dap-aggregate-share-req¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.7 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.7. "application/dap-aggregate-share" media type

Type name:

application¶

Subtype name:

dap-aggregate-share¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.7 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.8. "application/dap-collection-job-req" media type

Type name:

application¶

Subtype name:

dap-collection-job-req¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.7 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.1.9. "application/dap-collection-job-resp" media type

Type name:

application¶

Subtype name:

dap-collection-job-resp¶

Required parameters:

N/A¶

Optional parameters:

None¶

Encoding considerations:

only "8bit" or "binary" is permitted¶

Security considerations:

see Section 4.7 of the published specification¶

Interoperability considerations:

N/A¶

Published specification:

RFC XXXX¶

Applications that use this media type:

N/A¶

Fragment identifier considerations:

N/A¶

Additional information:

Magic number(s):

N/A¶

Deprecated alias names for this type:

N/A¶

File extension(s):

N/A¶

Macintosh file type code(s):

N/A¶

Person and email address to contact for further information:

see Authors' Addresses section of the published specification¶

Intended usage:

COMMON¶

Restrictions on usage:

N/A¶

Author:

see Authors' Addresses section of the published specification¶

Change controller:

IESG¶

9.2.1. Batch Modes Registry

A new registry will be (RFC EDITOR: change "will be" to "has been") created called "Batch Mode Identifiers" for DAP batch modes (Section 5). This registry should contain the following columns:¶

Value:

The one-byte identifier for the batch mode¶

Name:

The name of the batch mode¶

Reference:

Where the batch mode is defined¶

The initial contents of this registry listed in Table 2.¶

9.2.2. Report Extension Registry

A new registry will be (RFC EDITOR: change "will be" to "has been") created called "Report Extension Identifiers" for extensions to the upload interaction (Section 4.5). This registry should contain the following columns:¶

Value:

The two-byte identifier for the upload extension¶

Name:

The name of the upload extension¶

Reference:

Where the upload extension is defined¶

The initial contents of this registry are listed in Table 3.¶

9.2.3. Report Error Registry

A new registry will be (RFC EDITOR: change "will be" to "has been") created called "Report Error Identifiers" for reasons for rejecting reports during the aggregation interaction (Section 4.6.2.2).¶

Value:

The one-byte identifier of the report error¶

Name:

The name of the report error¶

Reference:

Where the report error is defined¶

The initial contents of this registry are listed below in Table 4.¶