Log Issue

• ident & loc data —

  nats sub -s nats://TOKEN@a15d11836d0644f6da0d09cbd81fae4f-949e37ea0352e6ad.elb.us-west-2.amazonaws.com:4222 "plane.ident.annotated"
  
  nats sub -s nats://TOKEN@a15d11836d0644f6da0d09cbd81fae4f-949e37ea0352e6ad.elb.us-west-2.amazonaws.com:4222 "plane.loc.annotated"
  

• How to get echo to work on Dockerfile to debug

  • RUN echo “hello”
  • while building add --progress=plain --no-cache
    • full command will look like docker build --progress=plain --no-cache -t nats-annotator .

• How to kill docker runs

  • docker ps
    • this allows your to see the running containers
  • docker stop <insert docker container id>

EC2 Error: Ran out of Space

• nats.errors.SlowConsumerError: nats: slow consumer, messages dropped subject

  • A slow consumer is a subscriber that cannot keep up with the message flow delivered from the NATS server. This is a common case in distributed systems because it is often easier to generate data than it is to process it. When consumers cannot process data fast enough, back pressure is applied to the rest of the system. NATS has mechanisms to reduce this back pressure.

  • Attempt 1: clearing messages in jetstream → works temporarily

    • Clearing Jetsteam
      • NATS command to clear it up
        • check the message and size for stream
          • nats -s nats://TOKEN@a15d11836d0644f6da0d09cbd81fae4f-949e37ea0352e6ad.elb.us-west-2.amazonaws.com:4222 stream ls
          • command output
        • purge the stream message and keep N messages
          • nats -s nats://TOKEN@a15d11836d0644f6da0d09cbd81fae4f-949e37ea0352e6ad.elb.us-west-2.amazonaws.com:4222 stream purge [STREAM_NAME] --keep [MSG_TO_KEEP]
          • command output

  • Attempt 2:

    • trying to see nats info in ec2 instance
      • https://comtechies.com/password-authentication-aws-ec2.html#:~:text=What is the default password,authentication for your ec2 instances.
      • downloading home brew https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-homebrew.html
      • install nats https://docs.nats.io/using-nats/nats-tools/nats_cli
      • try and use nats to debug https://docs.nats.io/running-a-nats-service/configuration/clustering/jetstream_clustering/troubleshooting
    • sudo git clone <https://aur.archlinux.org/yay.git>

  • Attempt 3: successfully installed nats

    • downloaded brew
    • downloaded go with brew
    • downloaded nats with go
    • go install github.com/nats-io/natscli/nats@latest
    • sudo find / -name nats

    

• Identifying this issue

  • Planes not showing up in front end → long/lat information not being passed → annotator out of space

  • in ec2 command line

    • sudo du -h --max-depth=1 too see which files on the machine are taking up the most space

    • if old Docker images are taking too much space, do: docker system prune -a -f ——— it’s important do this from time to time!

    • if still not enough space cleared (lets clear completley and restart docker)

      # from [<https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2>](<https://stackoverflow.com/questions/46672001/is-it-safe-to-clean-docker-overlay2>)
      # danger, read the entire text around this code before running
      # you will lose data
      
      sudo -s
      systemctl stop docker
      rm -rf /var/lib/docker
      systemctl start docker
      exit
      

      • sudo du -h --max-depth=1 /var/lib/docker

    • restart

      • Continue from steps under section “Run the annotator in EC2”(build and run docker image, screen, etc.)

    sudo du -h --max-depth=1 /var/lib/docker
    

    ---

    New Process to build/run on AWS:

    • SCP annotator files (but not data) → SCP to be replaced with Git
    • build script (build-docker-aws)
      • say “Y” when prompted about docker volumes prune
    • run script

    ---

    general var/lib/docker/overlay2 size at different points:

    • init (after docker reset) = 0

    • after image build = 422M

    • after docker run = 1.8G