Automated incident response · AWSΑργος Πανοπτης

Argus, automated incident response on AWS

When AWS raises a security finding, Argus saves the evidence, cuts the attacker off and tells a human. Median time to containment across six live incidents: 8 seconds.

Πανοπτηςall-seeingv1.1 · us-east-1 · Terraform
Fig. 1 · The watch

Argus Panoptes had a hundred eyes. Two slept at a time; the rest kept watch.

Fig. 2 · A finding

An instance starts resolving mining pools. GuardDuty notices. Every eye turns.

Fig. 3 · Contained

Eight seconds later it is isolated, its disks are snapshotted, and the evidence is sealed.

ΑThe watchphylax, watchmanΦΥΛΑΞ

Someone has to act on the finding.

AWS can tell you when a leaked key is used from a known-bad address, when an instance starts resolving crypto-mining pools, or when a bucket is read in bulk by someone who shouldn’t be reading it. It raises a finding, and the finding waits for a person.

Argus is the first ten seconds of that person’s response, written down. For each kind of attack there is a playbook that starts the moment the finding arrives and works in the order an incident responder would: preserve the evidence, contain the attacker, eradicate the access path, notify a human.

Most automation of this kind is a single function that fixes the resource. Fixing first destroys the record of what happened, and a failure halfway through leaves no trace. Argus runs every response as a state machine of separate, retryable phases, and every run either finishes or tells someone it didn’t.

Fig. 4 · Two clocks. One live IAM incident, to scalet 0:00.0
AWS

Attack → GuardDuty finding

≈ 9 min 21 s

remainder of 9 min 49 s

Security Hub, EventBridge

Finding → Argus

22.9 s

Argus

Argus → contained

5 s

×20

GuardDuty took nine and a half minutes to raise this finding. That clock belongs to AWS. Argus is judged on the second one, which starts when the finding reaches it: across six live incidents, containment took a median of 8 s and never more than 11 s, against a target of 60.

ΒThe relayphryktōria, beacon relayΦΡΥΚΤΩΡΙΑ

A fire, passed from hill to hill.

In Aeschylus’ Agamemnon, news that Troy has fallen reaches Argos in a single night, by beacon fires lit on one mountain after another. Each watcher only has to see the last fire and light the next.

Argus moves a finding the same way. Seven stations, each with one job, each handing a smaller and more exact message to the next.

  1. 01

    The attack

    attack_simulations/
    Receives
    An operator runs a script against deliberately weak resources, tagged argus:managed=true.
    Produces
    API calls from an address on a threat list, DNS lookups of mining pools, bulk reads of a bucket.
    shell
    python attack_simulations/simulate_ec2_mining.py
  2. 02

    Detection

    GuardDuty · AWS Config
    Receives
    CloudTrail, VPC Flow Logs and DNS logs, which GuardDuty reads on its own. Config watches the bucket's settings.
    Produces
    A finding with a type and a numeric severity: 7.0 to 8.9 is HIGH.
    finding
    CryptoCurrency:EC2/BitcoinTool.B!DNS
    minutes · AWS's clock
  3. 03

    Normalisation

    Security Hub
    Receives
    Findings from GuardDuty and Config.
    Produces
    The same finding in one format, ASFF, published as an event: Security Hub Findings - Imported.
    ASFF Types[0], from a captured event
    TTPs/Command and Control/CryptoCurrency:EC2-BitcoinTool.B!DNS
  4. 04

    Routing

    EventBridge
    Receives
    Every imported finding in the account.
    Produces
    One execution input, cut to ten fields. Five rules each match one finding type, their own minimum severity, active records only, and one exact simulation resource.
    pattern, argus-ec2
    "Types": [{ "wildcard": "*CryptoCurrency:EC2-BitcoinTool*" }],
    "Severity": { "Label": ["HIGH", "CRITICAL"] },
    "Resources": { "Id": ["arn:aws:ec2:…:instance/i-…"] }
    13–23 s · finding to Argus
  5. 05

    Orchestration

    Step Functions
    Receives
    The trimmed finding, which the first task turns into an incident envelope.
    Produces
    An ordered run of phases, each with its own timeout, retries and one shared failure path.
    three state machines, one template
    Validate → Record → Lock → Preserve → Contain
    → Eradicate → Package → Finalize → Notify
  6. 06

    Response

    Lambda · argus-remediation-role
    Receives
    The envelope, and the evidence collected before anything changed.
    Produces
    Changes to the resource, and a record of every call made.
    one recorded action
    { "action": "ec2:ModifyNetworkInterfaceAttribute",
      "target": "eni-…", "result": "ok", "at": "…" }
    median 8 s · finding to contained
  7. 07

    The record

    S3 Object Lock · DynamoDB · SNS · Discord
    Receives
    Every phase result.
    Produces
    A sealed evidence bundle with its SHA-256, the incident marked COMPLETE, the finding RESOLVED, a human told.
    the note Finalize writes on the finding
    Contained by Argus - incident f91bc10b42b2854b.
    Evidence: s3://argus-evidence-…/evidence/f91bc10b42b2854b/final_package.json
ΓThe ordertaxis, orderΤΑΞΙΣ

Preserve, contain, eradicate, notify.

Each playbook is a Step Functions state machine, and all three are rendered from one template. Two DynamoDB writes come first. They decide whether this run may touch anything at all.

After that, every phase is its own task with its own timeout and retries, so a failed notification is retried without containing the resource a second time. Any error in any phase lands on one path, and that path always reaches a human.

Replay an execution

A new finding on a free resource. Evidence first, then containment. Fifteen minutes later, once CloudTrail and Flow Logs have caught up, the late evidence is collected and sealed with its own digest.

Elapsed0.00 s
catch States.ALLconditional check failedresource lockedValidateFinding0.13 sRecordIncident0.14 sLockResource0.13 sPreserveEvidence3.7 sContain2.5 sEradicate1.6 sPackageEvidence2.1 sFinalize2.2 sNotify2.3 sWaitForLateLogs15 minCollectLateEvidenceDoneAlreadyHandledMarkDuplicateResolveDuplicateFindingChainActorS3DuplicateForResourceMarkFailedReleaseLockNotifyFailureRemediationFailedNotifyLateEvidence…
Lambda task Direct service call, no Lambda End state

Fig. 5 · Played at real speed with the per-phase medians of seven live runs. The fifteen-minute wait is fast-forwarded. A sum of medians is not a measured run: the measured median from finding to containment is 8 s.

ΔEvidencetekmērion, sure proofΤΕΚΜΗΡΙΟΝ

Before anything changes, write it down.

Containment destroys the before-state. Swap the security group and the old one is gone. Stop the instance and its memory is gone. Delete the bucket policy and the attacker’s policy is gone with it. So the first phase of every playbook records what the attacker could reach, and only then does anything change.

IAM user
  • The user and its groups
  • Console access
  • Every attached and inline policy
  • Each access key and its last use
  • The last hour of CloudTrail
  • Roles the user assumed
EC2 instance
  • Instance metadata
  • Security groups of every network interface
  • The instance profile
  • Subnet and network ACL
  • A snapshot of every EBS volume
S3 bucket
  • Bucket policy and its status
  • ACL and ownership controls
  • Block Public Access
  • Versioning
  • Every object version

The bundle is serialised canonically, with sorted keys and no whitespace, so the bytes stored are exactly the bytes hashed. Its SHA-256 sits beside it in sha256sum’s own format, and both go under S3 Object Lock: governance mode for a day while developing, or compliance mode for a year, which not even the root user can delete early.

Bundles verified
7 / 7
Late-evidence files
6 / 6
CloudTrail log files valid
49 / 49
Fig. 6 · Tamper with it

This is a bundle produced by Argus’s own packaging code. Change any character, or make the incident look one second faster, and watch the check fail.

evidence/f91bc10b42b2854b/final_package.json2,736 bytes
sha256.txt4a9ee2115e7dfdfd8467fb27dd8499e99682552fa3ffec43a8a6369c8db4451b final_package.json
$ sha256sum -c sha256.txt
final_package.json: OK
computed 4a9ee2115e7dfdfd8467fb27dd8499e99682552fa3ffec43a8a6369c8db4451b
Generated with build_bundle() and canonical_json() from the repository, in the shape of EC2 run 1. IDs are placeholders. The download buttons give you the exact bytes, so you can run sha256sum yourself.

ΤεκμηριονFor Aristotle, the kind of sign that admits no other reading. Rhetoric I.2

ΕPlaybooksstratēgēmata, stratagemsΣΤΡΑΤΗΓΗΜΑΤΑ

Three attacks, three playbooks.

Each one is aimed at a deliberately weak resource built by Terraform, and each is triggered by a real GuardDuty finding. The actions listed are the API calls the playbook makes, in order.

IAMIAM credential compromise

Leaked access key

MEDIUMUnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom

argus-sim-backdoor is a user whose access key sits in SSM, the way a leaked .env file would. The script uses that key from an address on a custom GuardDuty threat list.

Contain
  1. iam:UpdateAccessKeyDeactivate every access key
  2. iam:AttachUserPolicyAttach AWSDenyAll. The explicit deny also covers session tokens the user already minted
  3. iam:PutRolePolicyFor each role the user assumed, deny sessions issued before now (aws:TokenIssueTime)
Eradicate
  1. iam:DeleteLoginProfileRemove console access
  2. iam:TagUserTag QUARANTINED with the incident key

Keys are deactivated, never deleted. They are evidence.

Time to contain
5 s
Finding → Argus
22.9 s
Actions recorded
9
Late evidence
14 CloudTrail events

Verified liveThe backdoor user had assumed a role. Twelve minutes into that role session, after containment, its next call got an explicit deny.

EC2EC2 compromise

Crypto-mining

HIGHCryptoCurrency:EC2/BitcoinTool.B!DNS

argus-sim-miner is a t3.micro that resolves well-known mining-pool domains, run on the instance through SSM. No mining software is ever downloaded.

Contain
  1. ec2:ModifyNetworkInterfaceAttributeMove every network interface to the empty isolate group
  2. ec2:ReplaceNetworkAclAssociationPut a dedicated subnet behind the deny-all NACL. This is what cuts live sessions
  3. ec2:DisassociateIamInstanceProfileDetach the instance profile, and deny instance-role sessions issued before now
  4. ec2:ModifyInstanceAttributeTurn on termination protection
Eradicate
  1. ec2:StopInstancesStop, never terminate. Memory is lost; the disks were already snapshotted

Every volume is snapshotted in the evidence phase, before the first change.

Time to contain
8 · 10 · 11 s
Finding → Argus
15.6 s
Actions recorded
11
Late evidence
461–756 flow-log records

Verified liveRole credentials copied out of instance metadata worked from outside AWS before containment, and returned AccessDenied after it.

S3S3 exposure and bulk reads

Bucket exfiltration

HIGHDiscovery:S3/MaliciousIPCaller.Custom+ Policy:S3/BucketBlockPublicAccessDisabled · LOW+ Config: s3-bucket-public-read-prohibited · FAILED

argus-sim-data is a private bucket of fake PII. The script turns its Block Public Access off, then reads it in bulk as argus-sim-exfil from the threat-list address.

Contain
  1. s3:PutBucketPublicAccessBlockTurn all four Block Public Access flags back on
  2. s3:DeleteBucketPolicyDrop the public policy. The original is already in evidence
  3. s3:PutBucketAclSet the ACL private, if ACLs are enabled at all
Eradicate
  1. s3:PutBucketPolicyQuarantine policy: deny everyone except the remediation, auditor and deployer roles
  2. s3:PutBucketVersioningVersioning on, so nothing is silently overwritten
  3. states:StartExecutionStart the IAM playbook for the user who made the calls

Duplicate findings still chain the IAM playbook, so a second attacker credential is never left active.

Time to contain
6 s
Finding → Argus
13.2 s
Actions recorded
10
Late evidence
230 S3 data events

Verified liveAll ten of the attacker's later read attempts were denied.

ΖThe testelenchos, the testing that refutesΕΛΕΓΧΟΣ

A widely cited technique, tested and refuted.

Version 1.1 isolated an instance the way many guides suggest: move it to an allow-all security group so connection tracking lets go, then to an empty one. The plan said this would cut a live attacker off. Round two of testing checked.

A Session Manager session printing a timestamp every two seconds was held open from outside AWS while each method ran. Nothing was stopped, so the network control was the only variable.

Fig. 7 · A live session, held open from outside AWS, while each control is applied at 0 st −12 s
  1. Security groupSwap straight to the empty isolate groupSurvived 75 s+
  2. Security groupAllow-all “untrack” group, then isolate (the v1.1 playbook)Survived 75 s+
  3. Security groupUntrack, 30 s of traffic, then isolateSurvived 60 s+
  4. Network ACLDeny-all network ACL on the subnetCut within ~2 s
Each mark is a line the session printed (one every 2 s). Survival times are how long the session was watched after the change; it was still alive when observation ended. Time compressed for the replay.
Security group · stateful
packet     10.42.1.154:49822 → ssmmessages.us-east-1.amazonaws.com:443
conntrack  flow tracked when the session began  → allow
rules      not consulted for a tracked flow

A flow that was tracked when it started stays allowed whatever the rules become. The untrack step changes nothing for flows that already exist.

Network ACL · stateless
packet     10.42.1.154:49822 → ssmmessages.us-east-1.amazonaws.com:443
rules      * deny all  → deny
           every packet, every time

Every packet is judged again, so a deny takes effect at once. The playbook now moves dedicated subnets behind a pre-built deny-all NACL. On shared subnets it can’t, and relies on the stop instead.

The full playbook after the fix, on a real finding: NACL applied 23:51:20 · last line printed 23:51:21 · stop called 23:51:23. The NACL cut the session, not the stop.

Ten things only real AWS could show.

The unit tests use moto, which doesn’t enforce IAM and doesn’t rate-limit. They passed the whole time. Each of these appeared only against real AWS, and each one changed the design.

  1. 01

    Finding storms

    Seven findings for one instance started seven playbooks; six hit the EBS snapshot rate limit.

    Fix Per-resource lock, snapshot backoff

  2. 02

    Stopped instance replaced

    Stopping released the public IP, which Terraform read as drift and planned to replace the quarantined instance.

    Fix Ignore that attribute

  3. 03

    Recurrences ignored

    GuardDuty updates the same finding when activity recurs, so a repeat attack after recovery never triggered.

    Fix One incident per occurrence: ID + UpdatedAt

  4. 04

    Latency measured wrong

    Detection latency for a recurring finding was measured from its first creation.

    Fix Measure from UpdatedAt

  5. 05

    Attacker left active

    An admin call won the bucket lock, so the finding naming the real attacker was dropped as a duplicate.

    Fix Duplicates still chain the IAM playbook

  6. 06

    Untrack trick disproved

    Security-group swaps never cut a live session.

    Fix Deny-all NACL on dedicated subnets

  7. 07

    Missing permission

    ReplaceNetworkAclAssociation is also authorised against the subnet. The failure path caught it: four alerts.

    Fix Grant it on dedicated subnets

  8. 08

    Alert overstated damage

    A failure email said “NOT contained” when part of the step had already run.

    Fix List completed phases, warn on partial steps

  9. 09

    Alerts could go silent

    Changing the webhook parameter’s key would have re-put it with the disabled placeholder.

    Fix Ignore that attribute

  10. 10

    Slow on public buckets

    GuardDuty’s update took about ten minutes to arrive; AWS Config knew in under one.

    Fix Route the Config finding: 618 s → 218 s

ΗOncehapax, onceΑΠΑΞ

Every resource is contained once.

GuardDuty raised seven findings for one instance, one for each mining domain it resolved, and seven playbooks started at the same moment. Six of them hit the EBS snapshot rate limit.

Two conditional writes now stand in front of every playbook. The first is keyed on the finding occurrence, its ID plus UpdatedAt: an exact re-delivery stops there, but an attack that recurs after recovery is a new incident. The second locks the resource, so different findings about the same instance can’t race each other.

Each is a single DynamoDB call. There is no gap between checking and writing for a second execution to slip through, which a read followed by a write would leave open.

RecordIncident · ConditionExpressionkey: finding ID # UpdatedAt
attribute_not_exists(finding_id)
OR #status = :failed
LockResource · ConditionExpressionkey: the resource
attribute_not_exists(finding_id)
OR lock_until < :now
OR owner_finding_id = :me

A FAILED incident may run again: every phase is idempotent on its own, because Step Functions retries can run any task twice.

Fig. 8 · Every live execution, by where it stopped
Re-deliveries stopped
0 / 21
Duplicates stopped
0 / 18
Full runs
0 / 7
Resources contained twice
0
The counts are the live totals across all three playbooks; the order and spacing of arrivals are illustrative.

ΑπαξOnce. The same word as in hapax legomenon, a word that occurs only once in a text.

ΘSleeplessagrypnos, sleeplessΑΓΡΥΠΝΟΣ

Hermes killed Argus by lulling all hundred eyes to sleep at once, with a reed pipe and a long story.

Automated response fails the same way: quietly, with nobody told.

Nothing fails quietly.

  1. 01

    Every phase catches every error

    PreserveEvidence through Notify each carry Catch States.ALL, routed to one path.

  2. 02

    Timeouts are set per task

    When a whole execution times out, no Catch runs and the incident would sit IN_PROGRESS forever. Each task has its own TimeoutSeconds instead; a 1 s timeout was used to prove the catch fires.

  3. 03

    The lock is released

    So a retry, or the next finding about that resource, can act on it.

  4. 04

    The alert says what already happened

    It lists the phases that finished and warns that the failed step may have partly run.

  5. 05

    FAILED incidents run again

    The first conditional write lets a FAILED record be overwritten. A failed EC2 incident was retried after the fix and completed every phase.

6live failures6alert emails
Subject[Argus] FAILED: Crypto-mining on EC2 on i-0123456789abcdef0 - action required
ARGUS REMEDIATION FAILED: MANUAL ACTION REQUIRED

Crypto-mining on EC2 - severity HIGH
Finding   CryptoCurrency:EC2/BitcoinTool.B!DNS
Target    arn:aws:ec2:us-east-1:…:instance/i-0123456789abcdef0

WHAT WENT WRONG
  Failed step   PreserveEvidence
  Error         SnapshotCreationPerVolumeRateExceeded

COMPLETED BEFORE THE FAILURE
  no phase completed
  Steps inside PreserveEvidence may have partly run:
  check the target's current state.

NEXT STEPS
  1. Contain the target by hand if it is still exposed.
  2. Open the execution to see the failing step's input and error.
  3. Fix the cause. The incident is marked FAILED, so it runs
     again automatically the next time GuardDuty updates the finding.
Fig. 9 · The failure email, laid out by the notifier’s own template. Abridged; the same text goes to Discord as an embed.

A small blast radius.

The playbook functions hold almost no permissions. Each may assume one role, argus-remediation-role, and that role may only touch resources tagged argus:managed=true. A bug, or a hijacked function, can do no more than the playbook was meant to.

It is the cross-account pattern run inside one account. Moving to a separate security account changes an account ID in a role ARN and one trust policy.

IAM policy simulator · argus-remediation-role
  • allowRemediation actionson the simulation user, instance and bucket
  • denyAny remediation actionon a user outside the simulation
  • denyAny remediation actionon a bucket outside the simulation
  • denyAny remediation actionon an instance without argus:managed=true
  • denyec2:TerminateInstanceson the simulation instance itself
  • denyiam:AttachUserPolicywith any policy other than AWSDenyAll
Three customer-managed keys, one per data boundary, rotation on
  • argus-cloudtrail-key

    CloudTrail logs

    Only the trail may encrypt; late-evidence collection may decrypt.

  • argus-evidence-key

    Evidence bucket and Flow Logs

    Writers may encrypt; the packager and auditor may decrypt.

  • argus-ssm-key

    Webhook URL and simulation keys

    Only the notifier and the operator may decrypt.

ΙResultsapodeixis, demonstrationΑΠΟΔΕΙΞΙΣ

Every number here was read back from AWS.

Execution histories, the incidents table, the evidence bundles and the APIs themselves, from the live deployment in us-east-1 on 16 November 2026. Including the claims that are not verified yet.

Median time to contain
8s
Slowest of six
11s
Evidence digests valid
7/7
Resources contained twice
0
Failures nobody heard about
0
0 s10 s20 s30 s40 s50 s60 sspec target · under 60 smedian 8 s
Fig. 10 · Time to contain, each live incident: finding received to containment complete. Max 11 s.
Median per phase · seven full runsbar median · tick max
  1. ValidateFinding0.13 s
  2. RecordIncident0.14 s
  3. LockResource0.13 s
  4. PreserveEvidence3.70 s
  5. Contain2.50 s
  6. Eradicate1.60 s
  7. PackageEvidence2.10 s
  8. Finalize2.20 s
  9. Notify2.30 s

Preserving evidence is the slowest step. That is the price of collecting forensics before containment, and it is under four seconds.

The spec’s claims, checked● verified · ◐ partly · ○ not yet
  • ●Under 60 s to containmentMaximum 11 s across six live incidents
  • ●Phase order: preserve → contain → eradicate → notifyConfirmed in every full execution history
  • ●Idempotency guards21 re-deliveries and 18 duplicates stopped; no resource contained twice
  • ●Failure path alerts a human6 live failures, 6 FAILED emails
  • ●Task timeouts reach the failure path1 s task timeout: States.Timeout caught by the same Catch
  • ●Live EC2 connections are actually cutAfter the fix: deny-all NACL cuts a live session in ~1–2 s
  • ●Assumed-role sessions revoked12-minute-old role session refused with an explicit deny
  • ●Evidence bundles verifiable with sha256sum7 / 7 bundles, 6 / 6 supplemental files
  • ●CloudTrail log file validationvalidate-logs: 2 / 2 digest files, 49 / 49 log files
  • ●Remediation limited to simulation resourcesIAM policy simulator denies every out-of-scope call tried
  • ●100% of Argus in Terraform147 resources; no drift after all tests. IAM Identity Center and the SNS confirmation were manual
  • ◐Console password deletedLive runs correctly skipped it (no password set); deletion is unit-tested
  • ◐Budget guardrailAlarm created, never triggered; teardown not yet run
  • ○“Tens of dollars a month”Cost Explorer had no data yet. Recheck pending
ΚMaterialstechnē, craftΤΕΧΝΗ

What it is made of, and what it does not do.

The stack
Detection
GuardDuty (S3 Protection, custom threat list) · AWS Config · CloudTrail · VPC Flow Logs
Normalisation
Security Hub CSPM, findings in ASFF
Routing
EventBridge: five scoped rules with input transformers
Orchestration
Step Functions, one ASL template rendered three times
Response
Python 3.12 Lambdas (boto3) behind an assumed remediation role
State
DynamoDB conditional writes: the incident record and the resource lock
Evidence
S3 Object Lock · SHA-256 · Athena over CloudTrail · Logs Insights
Keys
Three KMS customer-managed keys, rotation on
Alerting
SNS email by severity · Discord webhook from SSM SecureString
Infrastructure
Terraform ≥ 1.10, remote state with S3 lockfile
Tests
pytest + moto, plus the three live simulations
What Argus does not do
  • No memory forensics

    Stopping the instance erases RAM. The disks are kept and snapshotted.

  • One region

    GuardDuty, Security Hub and Config run in us-east-1 only.

  • One account

    Remediation goes through an assumed role, so moving to a second account changes an ARN and a trust policy. It has not been done.

  • Recovery is manual

    Restoring groups, profiles and policies is a runbook step after review, on purpose.

  • Cost and teardown unverified

    Cost Explorer had no data yet, and terraform destroy has not been run.

Run it yourselfREADME ↗
cd terraform/bootstrap && terraform apply      # state bucket, once
cd .. && terraform init -backend-config=backend.hcl
terraform plan -out argus.tfplan && terraform apply argus.tfplan

python attack_simulations/simulate_ec2_mining.py     # findings in 5–15 min
python attack_simulations/reset_simulation.py --all --start-instance

sha256sum -c sha256.txt                              # verify any bundle