README.md

Kinesis Paraglider Challenge Team Hackathon Challenge

Use AWS's Kinesis service to find the optimal time to go Paragliding! We will be using weather data provided by NOAA with a provided event streamer.

Pre-requisites

The following tools and accounts are required to complete these instructions.

• Sign-up for an AWS account
• Install AWS CLI
• NodeJs
• Install .NET Core 2.1

Level 0

• Create a Kinesis stream in the AWS console (make sure the Region is set to US-East-1)
  • Search for Kinesis in the services dropdown
  • Click Create data steam under the Kinesis data streams section on the Amazon Kinesis Dashboard
  • Give it a name and 1 shard
• Clone this repo to get the data streamer application on your machine
• Update kinesis key name in src/WeatherStationsEvents/appsettings.json to match the name you just gave to your stream

Build and run 'src/WeatherStationsEvents'

• Go to terminal
• CD into the repo you just cloned
• CD into 'src/WeatherStationEvents'
• Run 'dotnet restore' and 'dotnet run'
• Verify from logs in the terminal that events are being generated
• NOTE: This data is historical. It starts from an arbitrary date in August 2018 and continues up to today.

Level 1

Goal - Create a lambda function to capture the streaming data from the Kinesis stream you just set up.

• Here is a ready to go NodeJs 6.10 lambda function for reading a Kinesis Stream. Make the lambda function output the data to CloudWatch Logs

exports.handler = (event, context, callback) => {
    
    for (let i = 0; i < event.Records.length; i++) {
      const eventRecord = JSON.parse(Buffer.from(event.Records[i].kinesis.data, 'base64'));
      console.log(eventRecord); 
    }
    
    callback(null, "Hello from Lambda");
};

Make sure to set up this function to trigger off of the Kinesis Stream

• Navigate to the AWS console for your lambda function
• Make sure the configuration tab is selected at the top of the page
• From the list of triggers on the left panel in the Designer, choose Kinesis
• Scroll down to the 'Configure triggers' section
• Select the Kinesis Stream you previously created from the dropdown
• Make sure the 'Enable trigger' box is checked, then hit 'Add'

Check CloudWatch logs for event record output

• Once the trigger is setup, run the streaming application from the terminal
• A record should be pushed to the Kinesis stream every five seconds and processed by your lambda function
• On the lambda function page, click the 'Monitoring' tab at the top and click the 'View logs in CloudWatch button on the right

Level 2 - Single Site Notification

Goal - Find the best time to go fly at Torrey Pines Gliderport. Analyze the streaming data and determine if the weather is good for paragliding.

• Conditions - If the conditions at Torrey Pines satisfy these, then it's good to fly!
  • Less than 80% humidity
  • Wind direction 230 to 290 degrees
  • Wind speed 6 to 12 knots
  • Gusts below 20 knots
• Trigger a message to CloudWatch logs to inform when conditions are good to fly
• Integrate AWS SNS to SMS service to send a notification to yourself
• Helpful docs
  • https://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html
  • https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sns-examples-publishing-messages.html

Hints

• Explore the event record object to find the attributes that need to be checked

Level 3 - Kinesis Data Analytics

Goal - Use Kinesis Data Analytics to add a lambda function for pre-processing records.

• Replace NA with values of zero on field barometricPressure
• Hook up the output of the pre processing to your original lambda function

Helpful docs

• https://docs.aws.amazon.com/kinesisanalytics/latest/dev/getting-started.html

Hints

• Be very careful with the IAM role for Data Analytics permissions
• Make sure the data streaming application is running when using DA
• Data is base64 encoded!

Level 4 - Multi Site Notification

Goal - Find the best time to go fly at more than one location.

• Use this website to determine the best conditions at another location or locations
  • https://www.sdhgpa.com/sites-guide.html
  • Choose one (or more) of the 5 sites under 'Primary Sites'
  • Look at the description of the site and find the recommended flying conditions at that site
• Update your lambda function to check the streaming data for flying conditions at multiple sites
• Update the function to:
  • Only send one notification per site per day
  • Only notify during daylight hours - 9 AM to 6 PM

Hints

• You will have to persist the data across lambda invocations in order to know if a notification has already been sent...

BOSS Level - Real Time Analytics with SQL

Goal - Use Kinesis Data Analytics to create a new data stream to better fit the original lambda function application.

• Within the Kinesis Data Analytics Application you created for step 3 use the SQL editor to perform real time analytics on the data
• Create a new SQL query using the templated SQL examples. Use the source data as a guide
• Attach the resulting stream of the real time analytics to your original lambda function