Skip to content

Commit 4fe8686

Browse files
author
Andrew Dampf
committed
initial commit
0 parents  commit 4fe8686

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+3386
-0
lines changed

‎.gitignore

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.env
2+
.coverage
3+
*.log
4+
__pycache__/
5+
venv/
6+
htmlcov/
7+
migrations/
8+
app.db

‎LICENSE

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright 2019 Linode
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎README.md

+262
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
janitor
2+
=======
3+
4+
Janitor is a flask application for parsing provider maintenance notification emails and taking actions based on those emails. It's written to be easily extensible to your environment.
5+
6+
7+
# Overview
8+
Janitor connects to an email server on a user-specified interval and checks for any maintenance emails from a list of providers and adds them to the database. It can then be configured to take an action based on the type of email: new, update, cancel, reschedule, started, and ended. For instance, you can post updates to slack on maintenance start/end emails, add events to your calendar for new emails, remove events from your calendar for cancelled emails, etc.
9+
10+
# Demo
11+
![demo](docs/demo.gif)
12+
13+
# Components
14+
15+
## Mail Clients
16+
A mail client is configured with an email address, password, and port so that it can be connected to for email retrieval. Currently only gmail is supported.
17+
18+
## Providers
19+
Currently supported providers include:
20+
- NTT *
21+
- PacketFabric *
22+
- EUNetworks *
23+
- GTT
24+
- Zayo
25+
26+
27+
The * Providers follow the [maint note](https://github.com/jda/maintnote-std/blob/master/standard.md) standard.
28+
29+
# Configuration Options
30+
31+
### PROJECT_ROOT
32+
The root folder of the janitor app. default: current working directory
33+
34+
### SECRET_KEY
35+
Your application's secret key. This is required.
36+
37+
### MAX_CONTENT_LENGTH
38+
Maximum size for circuit contract file uploads. default: 32 Mib
39+
40+
### LOGFILE
41+
Location of the file that logs are written to.
42+
43+
### CHECK_INTERVAL
44+
How frequently the mail server is checked for new messages. default: 10 minutes
45+
46+
### POSTS_PER_PAGE
47+
The number of maintenances/circuits/providers to display on a single page. default: 20
48+
49+
### DATABASE_URL
50+
The location of the database. all databases supported by sqlalchemy are supported. default: current working directory + app.db
51+
52+
### TZ_PREFIX
53+
For correctly modifying timezones. Some providers send maintenances with a timezone of "Eastern" instead of "US/Eastern" which breaks python datetime. You could set the TZ_PREFIX value to "US/" to fix this issue. default: None
54+
55+
### MAIL_USERNAME
56+
Username for your mail server. This is required
57+
58+
### MAIL_PASSWORD
59+
Password for your mail server. This is required
60+
61+
### MAIL_SERVER
62+
imap address of your mail server
63+
64+
### MAILBOX
65+
The name of the mailbox to process messages from. default: INBOX
66+
67+
### MAIL_CLIENT
68+
The mail client you wish to use. currently only gmail is supported. This is required.
69+
70+
### SLACK_WEBHOOK_URL
71+
If you wish to send messages to slack, you can define this. default: None
72+
73+
### SLACK_CHANNEL
74+
The channel to post slack messages to. default: None
75+
76+
77+
# database schema
78+
79+
![db schema](docs/schema.png)
80+
81+
# Setup
82+
Below walks through installation on ubuntu
83+
84+
## Database
85+
You can choose any database you'd like. The example below uses mariadb.
86+
1. install mariadb
87+
```
88+
apt install mariadb-server
89+
```
90+
2. install mariadb dependencies
91+
```
92+
apt install libmariadbclient-dev
93+
pip3 install mysqlclient
94+
```
95+
3. create the database
96+
```
97+
mariadb
98+
MariaDB [(none)]> create database janitor CHARACTER SET utf8;
99+
MariaDB [(none)]> CREATE USER 'janitor'@'localhost';
100+
MariaDB [(none)]> GRANT ALL PRIVILEGES ON janitor.* TO 'janitor'@'localhost';
101+
MariaDB [(none)]> quit
102+
103+
```
104+
105+
106+
## requirements
107+
janitor requires python3.6
108+
109+
1. clone this repository into your desired installation directory
110+
2. create your virtual environment and activate it
111+
```
112+
python3 -m venv venv
113+
source venv/bin/activate
114+
```
115+
3. install the requirements
116+
```
117+
pip3 install -r requirements.txt
118+
```
119+
4. create a `.env` configuration files with the necessary variables. For example:
120+
```
121+
PROJECT_ROOT='/opt/janitor'
122+
DATABASE_URL='mysql://janitor@localhost/janitor?charset=utf8'
123+
CHECK_INTERVAL=300
124+
SLACK_WEBHOOK_URL='https://hooks.slack.com/abc123'
125+
SLACK_CHANNEL='#mychannel'
126+
MAIL_USERNAME='user@example.com'
127+
MAIL_PASSWORD='mypassword'
128+
MAIL_SERVER='imap.example.com'
129+
MAIL_CLIENT='Gmail'
130+
SECRET_KEY='mysecretkey'
131+
```
132+
5. create the db schema:
133+
```
134+
flask db init
135+
flask db migrate
136+
flask db upgrade
137+
```
138+
6. You may wish to choose the providers you have in your network at this point, rather than selecting them all. You can do so by editing `app/jobs/main.py` and removing the ones you don't want in the `PROVIDERS` list
139+
7. From the providers you selected, you can define the email and type info under each provider's class in `app/Providers.py` types are one of: `transit`, `backbone`, `transport`, `peering`, and `facility`.
140+
8. at this point you can test connectivity to your server. First run the server:
141+
```
142+
flask run -h 0.0.0.0
143+
```
144+
and then open a browser to your IP to see if you can connect.
145+
146+
147+
## Web/uwsgi Server
148+
It's not recommended to expose the flask app directly to the internet. Below we'll use nginx and gunicorn with supervisord for setup. It's also recommended to use https, though the the steps below only cover http
149+
150+
1. install the packages
151+
```
152+
apt install nginx supervisor
153+
pip3 install gunicorn
154+
```
155+
2. create /etc/supervisor/conf.d/janitor.conf with the following contents:
156+
```
157+
[program:janitor]
158+
command=/opt/janitor/venv/bin/gunicorn -b localhost:8000 -w 4 janitor:app --preload
159+
directory=/opt/janitor
160+
user=root
161+
autostart=true
162+
autorestart=true
163+
stopasgroup=true
164+
killasgroup=true
165+
```
166+
3. create /etc/nginx/sites-enabled/janitor with the following contents:
167+
```
168+
server {
169+
# listen on port 80 (http)
170+
listen 80;
171+
server_name _;
172+
access_log /var/log/janitor_access.log;
173+
error_log /var/log/janitor_error.log;
174+
location / {
175+
# forward application requests to the gunicorn server
176+
proxy_pass http://localhost:8000;
177+
proxy_redirect off;
178+
proxy_set_header Host $host;
179+
proxy_set_header X-Real-IP $remote_addr;
180+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
181+
}
182+
183+
location /static {
184+
# handle static files directly, without forwarding to the application
185+
alias /opt/janitor/app/static;
186+
expires 30d;
187+
}
188+
}
189+
```
190+
4. restart nginx and supervisor
191+
```
192+
supervisorctl reload janitor
193+
systemctl restart nginx
194+
```
195+
196+
janitor only checks for unread messages in your inbox. You may want to mark a few messages as unread to see if messages are being parsed at this point by navigating to your janitor server's IP, and either waiting until the next email check runs (it will tell you when this will be), or by using the "process emails" button to process immediately. Once the connection is successful you can stop the server with Ctrl-C
197+
198+
# API
199+
The API has a UI and can be reached via the `/api/v1/ui/` endpoint for testing.
200+
201+
## Endpoints
202+
All API endpoints need to be prefaced with `/api/v1`, for example, `/api/v1/circuits`
203+
204+
### /circuits
205+
#### GET
206+
Get all circuits
207+
eg:
208+
`curl -X GET --header 'Accept: application/json' 'https://192.0.2.1/api/v1/circuits'`
209+
210+
#### POST
211+
Create a new circuit by posting json.
212+
eg:
213+
```
214+
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ \
215+
"a_side": "string", \
216+
"id": 0, \
217+
"provider_cid": "string", \
218+
"provider_id": 0, \
219+
"z_side": "string" \
220+
}' 'http://127.0.0.1:5000/api/v1/circuits'
221+
```
222+
223+
### /circuits/{circuit_id}
224+
#### GET
225+
Get a single circuit by the ID
226+
eg:
227+
`curl -X GET --header 'Accept: application/json' 'http://127.0.0.1:5000/api/v1/circuits/1'`
228+
229+
#### PUT
230+
Update a circuit by posting json
231+
eg:
232+
```
233+
curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{ \
234+
"a_side": "string", \
235+
"provider_id": 1 \
236+
}' 'http://127.0.0.1:5000/api/v1/circuits/5'
237+
```
238+
239+
### /maintenances
240+
#### GET
241+
Get all maintenances
242+
eg:
243+
`curl -X GET --header 'Accept: application/json' 'http://127.0.0.1:5000/api/v1/maintenances'`
244+
245+
### /maintenances/{maintenance_id}
246+
#### GET
247+
Get a maintenance by id
248+
eg:
249+
`curl -X GET --header 'Accept: application/json' 'http://127.0.0.1:5000/api/v1/maintenances/1'`
250+
251+
### /providers
252+
#### GET
253+
Get all providers
254+
eg:
255+
`curl -X GET --header 'Accept: application/json' 'http://127.0.0.1:5000/api/v1/providers'`
256+
257+
### /providers/{provider_id}
258+
#### GET
259+
Get a provider by id
260+
eg:
261+
`curl -X GET --header 'Accept: application/json' 'http://127.0.0.1:5000/api/v1/providers/1'`
262+

‎api/v1/circuits.py

+92
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
from app.models import Circuit, CircuitSchema, Provider
2+
from flask import make_response, jsonify
3+
from app import db
4+
5+
6+
def read_all():
7+
"""
8+
This function responds to a request for /circuits
9+
with the complete lists of circuits
10+
11+
:return: sorted list of circuits
12+
"""
13+
circuits = Circuit.query.all()
14+
schema = CircuitSchema(many=True)
15+
16+
return schema.dump(circuits).data
17+
18+
def read_one(circuit_id):
19+
circuit = Circuit.query.filter(Circuit.id == circuit_id).one_or_none()
20+
21+
if not circuit:
22+
text = f'circuit not found for id {circuit_id}'
23+
return make_response(jsonify(error=404, message=text), 404)
24+
25+
schema = CircuitSchema()
26+
data = schema.dump(circuit).data
27+
28+
return data
29+
30+
def create(circuit):
31+
"""
32+
creates a circuit! checks to see if the provider_cid is unique and
33+
that the provider exists.
34+
35+
:return: circuit
36+
"""
37+
provider_cid = circuit.get('provider_cid')
38+
provider_id = circuit.get('provider_id')
39+
circuit_exists = (
40+
Circuit.query.filter(Circuit.provider_cid == provider_cid)
41+
.one_or_none()
42+
)
43+
44+
provider_exists = (
45+
Provider.query.filter(Provider.id == provider_id)
46+
.one_or_none()
47+
)
48+
49+
if circuit_exists:
50+
text = f'Circuit {provider_cid} already exists'
51+
return make_response(jsonify(error=409, message=text), 409)
52+
53+
54+
if not provider_exists:
55+
text = (f'Provider {provider_id} does not exist.'
56+
'Unable to create circuit')
57+
return make_response(jsonify(error=403, message=text), 403)
58+
59+
60+
61+
schema = CircuitSchema()
62+
new_circuit = schema.load(circuit, session=db.session).data
63+
64+
db.session.add(new_circuit)
65+
db.session.commit()
66+
67+
data = schema.dump(new_circuit).data
68+
69+
return data, 201
70+
71+
72+
def update(circuit_id, circuit):
73+
"""
74+
updates a circuit!
75+
:return: circuit
76+
"""
77+
c = Circuit.query.filter_by(id=circuit_id).one_or_none()
78+
if not c:
79+
text = f'Can not update a circuit that does not exist!'
80+
return make_response(jsonify(error=409, message=text), 404)
81+
82+
83+
84+
schema = CircuitSchema()
85+
update = schema.load(circuit, session=db.session).data
86+
87+
db.session.merge(update)
88+
db.session.commit()
89+
90+
data = schema.dump(c).data
91+
92+
return data, 201

0 commit comments

Comments
 (0)