How to Export Jenkins Audit Logs to S3 for Your SIEM

What Jenkins records on its own

Jenkins has system logs, build logs and an access log you can turn on, but none of them is a structured audit trail of who changed what. Most teams add a plugin for that. The free Audit Trail plugin records job and view changes, builds started and deleted, script console use, credential use, logins, failed logins and logouts, and writes them to files on disk, syslog, the console or Elasticsearch. It doesn't write to S3, so with it you ship its files to the bucket with your own agent.

The worked steps below use the S3 export of the Audit Vault plugin, which uploads its audit events to S3 itself. The bucket, permissions and SIEM steps are the same whichever tool writes the objects; only the object format changes.

Four things that can fail separately

1. Capture: Jenkins recorded the action as an audit event.
2. Delivery: the event reached an object in the bucket.
3. Preservation: nobody could change or delete that object afterwards.
4. Indexing: the SIEM parsed the event, with the right time, and you can search for it.

Each one needs its own check. A valid hash chain says nothing about events that were never captured, and an object in S3 says nothing about whether the SIEM parsed it.

Step 1: Decide what you keep, and for how long

Write down which event categories you need (logins, job and build changes, credential use, system changes) and three retention periods, because they are set in three places:

• On the controller. Audit Vault keeps events in a local database and deletes one only after every active export destination has confirmed it, then waits the configured retention (90 days by default).
• In the bucket. S3 Lifecycle rules, versioning and Object Lock.
• In the SIEM. Its own index retention, usually much shorter.

If you move old objects to an archive storage class, they have to be restored before anything can read them again, so plan for replaying them into the SIEM.

Audit events name users, IP addresses, jobs and credentials. Treat the bucket and the SIEM index as sensitive.

Step 2: Create the bucket

• Keep Block Public Access on.
• Turn on versioning, so an overwrite or delete keeps the earlier version.
• Encryption: S3 encrypts new objects with S3-managed keys (SSE-S3) by default. Audit Vault's Server-Side Encryption box asks for exactly that (SSE-S3). If the bucket's default is a KMS key (SSE-KMS), leave the box unchecked so the bucket default applies, and give the writer kms:GenerateDataKey and the SIEM reader kms:Decrypt on the key (AWS SSE-KMS documentation).
• Object Lock (optional): Object Lock needs versioning and protects individual object versions. Audit Vault sets no retention on its uploads, so only a default retention rule on the bucket protects them. Governance mode can be bypassed by anyone with s3:BypassGovernanceRetention; compliance mode can't be shortened or removed by anyone, including the root user, until it expires. Even locked, a new version or a delete marker can still be added on top. We haven't tested Audit Vault's uploads against a bucket with default retention, so upload to a test bucket with the same settings and confirm objects arrive before you rely on it.

Step 3: Give Jenkins a write-only identity

Audit Vault's S3 export signs in with an Access Key ID and Secret Access Key only: it doesn't use instance roles or temporary credentials. Create an IAM user just for this, with a policy that can write to the prefix and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteAuditObjects",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::example-jenkins-audit/jenkins/audit/*"
    },
    {
      "Sid": "OptionalConnectionTest",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::example-jenkins-audit"
    }
  ]
}

The upload only needs s3:PutObject. The second statement is optional: the Test S3 Connection button checks that the bucket exists, and AWS requires s3:ListBucket for that check. Without it the test fails while exports still work. With it the test can pass even if the upload is denied. Either way, the real test is an object in the bucket (step 5).

The writer shouldn't be able to read, delete or change bucket settings. Rotate its access key on a schedule, and keep separate identities for the SIEM reader (step 6) and for whoever administers the bucket.

Step 4: Turn on the export in Jenkins

In Manage Jenkins → System, in the S3 Export section, check Enable S3 Export, then under S3 Configuration fill in Bucket Name, Region, Path Prefix (default jenkins/audit/) and the access key. Export Interval (minutes) defaults to 60 and applies to every export destination. Save.

Each run uploads up to 20 batches of 500 events. Events beyond that wait for the next run, so if you record more than 10,000 events per interval, shorten the interval or the backlog grows.

Step 5: Check an object

After the next export, list the prefix and download one object with its metadata:

aws s3 ls s3://example-jenkins-audit/jenkins/audit/ --recursive
aws s3api get-object --bucket example-jenkins-audit --key <key> object.ndjson > object.response.json

• Key: jenkins/audit/yyyy/MM/dd/HH/<controller>-<destination>-audit-<first seq>-<last seq>-<token>.ndjson. The date folder is the UTC hour of the batch's first event. A batch can span hours, so don't treat the folder as the time range of every event in it.
• Body: one JSON object per line (NDJSON, content type application/x-ndjson), with fields such as id, timestamp, eventType, actor, resourceName, summary, checksum and prevHash. Empty values are null.
• Metadata: four auditvault-* entries, the batch's sequence range and the audit chain head when it was exported, which the plugin's archive check uses.

A retry overwrites the same key, but after a restart or a configuration save a retry can write a second object that overlaps the first. Deduplicate on the controller plus the event id, and flag two copies of one event with different checksums.

Step 6: Tell the SIEM collector about new objects

Most S3 collectors read new objects from an SQS queue rather than listing the bucket:

1. Create a standard SQS queue in the bucket's region, with a dead-letter queue for messages the collector can't process.
2. Let the bucket send to the queue. A new queue doesn't accept messages from S3, so add this to the queue's access policy, with your queue, bucket and account:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": { "Service": "s3.amazonaws.com" },
         "Action": "sqs:SendMessage",
         "Resource": "arn:aws:sqs:us-east-1:111122223333:jenkins-audit-siem",
         "Condition": {
           "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::example-jenkins-audit" },
           "StringEquals": { "aws:SourceAccount": "111122223333" }
         }
       }
     ]
   }

   If the queue is encrypted with a customer-managed KMS key, also add s3.amazonaws.com to that key's policy with kms:GenerateDataKey and kms:Decrypt, as the AWS permissions page shows.
3. Add an S3 event notification for s3:ObjectCreated:* on the jenkins/audit/ prefix, sending to that queue.
4. Give the collector its own read-only identity. This policy fits collectors whose docs ask only for these actions, such as Filebeat's aws-s3 input; check your collector's own list:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "s3:GetObject",
         "Resource": "arn:aws:s3:::example-jenkins-audit/jenkins/audit/*"
       },
       {
         "Effect": "Allow",
         "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:ChangeMessageVisibility"],
         "Resource": "arn:aws:sqs:us-east-1:111122223333:jenkins-audit-siem"
       }
     ]
   }

   Splunk's SQS-based S3 input asks for more: s3:GetObjectVersion because step 2 turned versioning on, and further SQS actions. Use this policy for it instead, with your own queue and dead-letter queue ARNs in place of the examples, plus kms:Decrypt if the bucket uses a KMS key:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "SplunkReadVersionedObjects",
         "Effect": "Allow",
         "Action": ["s3:GetObject", "s3:GetObjectVersion"],
         "Resource": "arn:aws:s3:::example-jenkins-audit/jenkins/audit/*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "sqs:GetQueueUrl",
           "sqs:ReceiveMessage",
           "sqs:SendMessage",
           "sqs:DeleteMessage",
           "sqs:ChangeMessageVisibility",
           "sqs:GetQueueAttributes"
         ],
         "Resource": [
           "arn:aws:sqs:us-east-1:111122223333:jenkins-audit-siem",
           "arn:aws:sqs:us-east-1:111122223333:jenkins-audit-siem-dlq"
         ]
       },
       {
         "Effect": "Allow",
         "Action": "sqs:ListQueues",
         "Resource": "*"
       }
     ]
   }

S3 event notifications are delivered at least once and not in order, and can occasionally repeat, so the collector will see duplicates. Two consumers reading one queue split the messages between them, so give each SIEM its own queue. Notifications only cover new objects: load the objects already in the bucket separately.

Step 7: Connect your SIEM

No SIEM reads Audit Vault's objects just because they are in S3. Each needs a collector, a parsing rule and a timestamp rule:

Step 8: Prove it end to end

1. Do something harmless that is audited, such as creating and deleting a test job, and note the time.
2. Find the event on the Audit Vault dashboard in Jenkins (by time, actor and summary).
3. After the next export, find the same event in the downloaded object and note its id.
4. Search the SIEM for the same id and check the actor, event type and time match.

Then test the failure cases once: a denied upload (remove the writer's permission briefly and confirm the dashboard's export status shows the failure and catches up afterwards), a collector outage (stop it, then confirm the queue backlog drains without gaps), and a duplicate notification.

Step 9: Watch for gaps

• Alert when no new object has arrived for longer than two export intervals.
• Alert on the age of the oldest SQS message and on anything in the dead-letter queue.
• Alert when the SIEM has had no Jenkins events for a working day.
• Watch the controller's disk: events that can't be exported stay in the local database, and its size setting is a warning, not a limit.

Where Audit Vault fits

The Audit Vault plugin for Jenkins captures authentication, job, build, credential and system events, stores them in a local database with a SHA-256 hash chain, and exports them on a schedule. What it does, and its limits:

• Destinations: S3 and S3-compatible stores, a filesystem directory, SFTP, Azure Blob Storage, Google Cloud Storage, Splunk HEC, syslog and Datadog. The direct Splunk and Datadog exports skip S3, but an accepted request still doesn't prove the event was indexed.
• Capture has limits. Events wait in a 5,000-event queue before they are written; if it fills, new events are dropped and logged. Credential use is recorded at most once per credential and resource every five minutes. Disabled categories, a missing license and a crash before an event is written are also gaps.
• Checking an archive: the plugin's archive check compares an exported object with a copy of the controller's database and checks the recorded chain head. It doesn't prove the archive is complete, or protect against someone who can rewrite both the bucket and the database.
• Changing the bucket or prefix doesn't copy earlier events to the new location.

The settings are in the S3 Export and Event Capture sections of the docs, the hash chain in Integrity, and the license in Licensing.

Next steps

Read the full Audit Vault documentation for every setting, or see what the plugin includes and start a free trial on the Audit Vault product page.