STRLCPY/changedetection.io

■ ■ ■ ■ ■ ■

README-pip.md

1		-	# changedetection.io
2		-	![changedetection.io](https://github.com/dgtlmoon/changedetection.io/actions/workflows/test-only.yml/badge.svg?branch=master)
3		-	<a href="https://hub.docker.com/r/dgtlmoon/changedetection.io" target="_blank" title="Change detection docker hub">
4		-	<img src="https://img.shields.io/docker/pulls/dgtlmoon/changedetection.io" alt="Docker Pulls"/>
5		-	</a>
6		-	<a href="https://hub.docker.com/r/dgtlmoon/changedetection.io" target="_blank" title="Change detection docker hub">
7		-	<img src="https://img.shields.io/github/v/release/dgtlmoon/changedetection.io" alt="Change detection latest tag version"/>
8		-	</a>
	1	+	## Web Site Change Detection, Monitoring and Notification.
9	2
10		-	## Self-hosted open source change monitoring of web pages.
11		-
12		-	_Know when web pages change! Stay ontop of new information!_
	3	+	Live your data-life pro-actively, track website content changes and receive notifications via Discord, Email, Slack, Telegram and 70+ more
13	4
14		-	Live your data-life pro-actively instead of re-actively, do not rely on manipulative social media for consuming important information.
	5	+	[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring" title="Self-hosted web page change monitoring" />](https://lemonade.changedetection.io/start?src=pip)
15	6
16	7
17		-	<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring" title="Self-hosted web page change monitoring" />
18		-
19		-
20		-	Get your own private instance now! Let us host it for you!
21		-
22		-	[Try our $6.99/month subscription - unlimited checks, watches and notifications!](https://lemonade.changedetection.io/start), choose from different geographical locations, let us handle everything for you.
23		-
	8	+	[Don't have time? Let us host it for you! try our extremely affordable subscription use our proxies and support!](https://lemonade.changedetection.io/start)
24	9
25	10
26	11		#### Example use cases
27	12
28		-	Know when ...
29		-
30		-	- Government department updates (changes are often only on their websites)
31		-	- Local government news (changes are often only on their websites)
	13	+	- Products and services have a change in pricing
	14	+	- _Out of stock notification_ and _Back In stock notification_
	15	+	- Governmental department updates (changes are often only on their websites)
32	16		- New software releases, security advisories when you're not on their mailing list.
33	17		- Festivals with changes
34	18		- Realestate listing changes
	19	+	- Know when your favourite whiskey is on sale, or other special deals are announced before anyone else
35	20		- COVID related news from government websites
	21	+	- University/organisation news from their website
36	22		- Detect and monitor changes in JSON API responses
37		-	- API monitoring and alerting
	23	+	- JSON API monitoring and alerting
	24	+	- Changes in legal and other documents
	25	+	- Trigger API calls via notifications when text appears on a website
	26	+	- Glue together APIs using the JSON filter and JSON notifications
	27	+	- Create RSS feeds based on changes in web content
	28	+	- Monitor HTML source code for unexpected changes, strengthen your PCI compliance
	29	+	- You have a very sensitive list of URLs to watch and you do _not_ want to use the paid alternatives. (Remember, _you_ are the product)
38	30
39		-	Get monitoring now!
	31	+	_Need an actual Chrome runner with Javascript support? We support fetching via WebDriver and Playwright!</a>_
	32	+
	33	+	#### Key Features
	34	+
	35	+	- Lots of trigger filters, such as "Trigger on text", "Remove text by selector", "Ignore text", "Extract text", also using regular-expressions!
	36	+	- Target elements with xPath and CSS Selectors, Easily monitor complex JSON with JsonPath rules
	37	+	- Switch between fast non-JS and Chrome JS based "fetchers"
	38	+	- Easily specify how often a site should be checked
	39	+	- Execute JS before extracting text (Good for logging in, see examples in the UI!)
	40	+	- Override Request Headers, Specify `POST` or `GET` and other methods
	41	+	- Use the "Visual Selector" to help target specific elements
	42	+
40	43
41	44		```bash
42	45		$ pip3 install changedetection.io
		skipped 7 lines
50	53
51	54
52	55		Then visit http://127.0.0.1:5000 , You should now be able to access the UI.
53		-
54		-	### Features
55		-	- Website monitoring
56		-	- Change detection of content and analyses
57		-	- Filters on change (Select by CSS or JSON)
58		-	- Triggers (Wait for text, wait for regex)
59		-	- Notification support
60		-	- JSON API Monitoring
61		-	- Parse JSON embedded in HTML
62		-	- (Reverse) Proxy support
63		-	- Javascript support via WebDriver
64		-	- RaspberriPi (arm v6/v7/64 support)
65	56
66	57		See https://github.com/dgtlmoon/changedetection.io for more information.
67	58
		skipped 1 lines

■ ■ ■ ■ ■ ■

README.md

		skipped 1 lines
2	2
3	3		Live your data-life pro-actively, track website content changes and receive notifications via Discord, Email, Slack, Telegram and 70+ more
4	4
5		-	[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring" title="Self-hosted web page change monitoring" />](https://lemonade.changedetection.io/start)
	5	+	[<img src="https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/docs/screenshot.png" style="max-width:100%;" alt="Self-hosted web page change monitoring" title="Self-hosted web page change monitoring" />](https://lemonade.changedetection.io/start?src=github)
6	6
7	7		[![Release Version][release-shield]][release-link] [![Docker Pulls][docker-pulls]][docker-link] [![License][license-shield]](LICENSE.md)
8	8
		skipped 203 lines

■ ■ ■ ■ ■ ■

changedetectionio/__init__.py

1	1		#!/usr/bin/python3
2	2
3		-
4		-	# @todo logging
5		-	# @todo extra options for url like , verify=False etc.
6		-	# @todo enable https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl as option?
7		-	# @todo option for interval day/6 hour/etc
8		-	# @todo on change detected, config for calling some API
9		-	# @todo fetch title into json
10		-	# https://distill.io/features
11		-	# proxy per check
12		-	# - flask_cors, itsdangerous,MarkupSafe
13		-
14	3		import datetime
15	4		import os
16	5		import queue
		skipped 27 lines
44	33		from changedetectionio import html_tools
45	34		from changedetectionio.api import api_v1
46	35
47		-	__version__ = '0.39.18'
	36	+	__version__ = '0.39.19.1'
48	37
49	38		datastore = None
50	39
		skipped 502 lines
553	542		default = deepcopy(datastore.data['watching'][uuid])
554	543
555	544		# Show system wide default if nothing configured
556		-	if datastore.data['watching'][uuid]['fetch_backend'] is None:
557		-	default['fetch_backend'] = datastore.data['settings']['application']['fetch_backend']
558		-
559		-	# Show system wide default if nothing configured
560	545		if all(value == 0 or value == None for value in datastore.data['watching'][uuid]['time_between_check'].values()):
561	546		default['time_between_check'] = deepcopy(datastore.data['settings']['requests']['time_between_check'])
562	547
		skipped 35 lines
598	583		if form.fetch_backend.data == datastore.data['settings']['application']['fetch_backend']:
599	584		extra_update_obj['fetch_backend'] = None
600	585
601		-	# Notification URLs
602		-	datastore.data['watching'][uuid]['notification_urls'] = form.notification_urls.data
603	586
604		-	# Ignore text
	587	+	# Ignore text
605	588		form_ignore_text = form.ignore_text.data
606	589		datastore.data['watching'][uuid]['ignore_text'] = form_ignore_text
607	590
		skipped 47 lines
655	638		watch=datastore.data['watching'][uuid],
656	639		form=form,
657	640		has_empty_checktime=using_default_check_time,
	641	+	has_default_notification_urls=True if len(datastore.data['settings']['application']['notification_urls']) else False,
658	642		using_global_webdriver_wait=default['webdriver_delay'] is None,
659	643		current_base_url=datastore.data['settings']['application']['base_url'],
660	644		emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False),
	645	+	settings_application=datastore.data['settings']['application'],
661	646		visualselector_data_is_ready=visualselector_data_is_ready,
662	647		visualselector_enabled=visualselector_enabled,
663	648		playwright_enabled=os.getenv('PLAYWRIGHT_DRIVER_URL', False)
		skipped 23 lines
687	672		form = forms.globalSettingsForm(formdata=request.form if request.method == 'POST' else None,
688	673		data=default
689	674		)
	675	+
	676	+	# Remove the last option 'System default'
	677	+	form.application.form.notification_format.choices.pop()
	678	+
690	679		if datastore.proxy_list is None:
691	680		# @todo - Couldn't get setattr() etc dynamic addition working, so remove it instead
692	681		del form.requests.form.proxy
		skipped 39 lines
732	721		current_base_url = datastore.data['settings']['application']['base_url'],
733	722		hide_remove_pass=os.getenv("SALTED_PASS", False),
734	723		api_key=datastore.data['settings']['application'].get('api_access_token'),
735		-	emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False))
	724	+	emailprefix=os.getenv('NOTIFICATION_MAIL_BUTTON_PREFIX', False),
	725	+	settings_application=datastore.data['settings']['application'])
736	726
737	727		return output
738	728
		skipped 460 lines
1199	1189		datastore.delete(uuid.strip())
1200	1190		flash("{} watches deleted".format(len(uuids)))
1201	1191
1202		-	if (op == 'pause'):
	1192	+	elif (op == 'pause'):
1203	1193		for uuid in uuids:
1204	1194		uuid = uuid.strip()
1205	1195		if datastore.data['watching'].get(uuid):
		skipped 1 lines
1207	1197
1208	1198		flash("{} watches paused".format(len(uuids)))
1209	1199
1210		-	if (op == 'unpause'):
	1200	+	elif (op == 'unpause'):
1211	1201		for uuid in uuids:
1212	1202		uuid = uuid.strip()
1213	1203		if datastore.data['watching'].get(uuid):
1214	1204		datastore.data['watching'][uuid.strip()]['paused'] = False
1215	1205		flash("{} watches unpaused".format(len(uuids)))
	1206	+
	1207	+	elif (op == 'mute'):
	1208	+	for uuid in uuids:
	1209	+	uuid = uuid.strip()
	1210	+	if datastore.data['watching'].get(uuid):
	1211	+	datastore.data['watching'][uuid.strip()]['notification_muted'] = True
	1212	+	flash("{} watches muted".format(len(uuids)))
	1213	+
	1214	+	elif (op == 'unmute'):
	1215	+	for uuid in uuids:
	1216	+	uuid = uuid.strip()
	1217	+	if datastore.data['watching'].get(uuid):
	1218	+	datastore.data['watching'][uuid.strip()]['notification_muted'] = False
	1219	+	flash("{} watches un-muted".format(len(uuids)))
	1220	+
	1221	+	elif (op == 'notification-default'):
	1222	+	from changedetectionio.notification import (
	1223	+	default_notification_format_for_watch
	1224	+	)
	1225	+	for uuid in uuids:
	1226	+	uuid = uuid.strip()
	1227	+	if datastore.data['watching'].get(uuid):
	1228	+	datastore.data['watching'][uuid.strip()]['notification_title'] = None
	1229	+	datastore.data['watching'][uuid.strip()]['notification_body'] = None
	1230	+	datastore.data['watching'][uuid.strip()]['notification_urls'] = []
	1231	+	datastore.data['watching'][uuid.strip()]['notification_format'] = default_notification_format_for_watch
	1232	+	flash("{} watches set to use default notification settings".format(len(uuids)))
1216	1233
1217	1234		return redirect(url_for('index'))
1218	1235
		skipped 221 lines

■ ■ ■ ■ ■ ■

changedetectionio/forms.py

		skipped 313 lines
314	314
315	315		# Common to a single watch and the global settings
316	316		class commonSettingsForm(Form):
317		-
318		-	notification_urls = StringListField('Notification URL list', validators=[validators.Optional(), ValidateNotificationBodyAndTitleWhenURLisSet(), ValidateAppRiseServers()])
319		-	notification_title = StringField('Notification title', default=default_notification_title, validators=[validators.Optional(), ValidateTokensList()])
320		-	notification_body = TextAreaField('Notification body', default=default_notification_body, validators=[validators.Optional(), ValidateTokensList()])
321		-	notification_format = SelectField('Notification format', choices=valid_notification_formats.keys(), default=default_notification_format)
	317	+	notification_urls = StringListField('Notification URL list', validators=[validators.Optional(), ValidateAppRiseServers()])
	318	+	notification_title = StringField('Notification title', validators=[validators.Optional(), ValidateTokensList()])
	319	+	notification_body = TextAreaField('Notification body', validators=[validators.Optional(), ValidateTokensList()])
	320	+	notification_format = SelectField('Notification format', choices=valid_notification_formats.keys())
322	321		fetch_backend = RadioField(u'Fetch method', choices=content_fetcher.available_fetchers(), validators=[ValidateContentFetcherIsReady()])
323	322		extract_title_as_title = BooleanField('Extract <title> from document and use as watch title', default=False)
324		-	webdriver_delay = IntegerField('Wait seconds before extracting text', validators=[validators.Optional(), validators.NumberRange(min=1, message="Should contain one or more seconds")] )
	323	+	webdriver_delay = IntegerField('Wait seconds before extracting text', validators=[validators.Optional(), validators.NumberRange(min=1,
	324	+	message="Should contain one or more seconds")])
325	325
326	326		class watchForm(commonSettingsForm):
327	327
		skipped 27 lines
355	355		filter_failure_notification_send = BooleanField(
356	356		'Send a notification when the filter can no longer be found on the page', default=False)
357	357
358		-	notification_use_default = BooleanField('Use default/system notification settings', default=True)
	358	+	notification_muted = BooleanField('Notifications Muted / Off', default=False)
359	359
360	360		def validate(self, **kwargs):
361	361		if not super().validate():
		skipped 49 lines

■ ■ ■ ■ ■ ■

changedetectionio/model/Watch.py

		skipped 5 lines
6	6		mtable = {'seconds': 1, 'minutes': 60, 'hours': 3600, 'days': 86400, 'weeks': 86400 * 7}
7	7
8	8		from changedetectionio.notification import (
9		-	default_notification_body,
10		-	default_notification_format,
11		-	default_notification_title,
	9	+	default_notification_format_for_watch
12	10		)
13	11
14	12
		skipped 17 lines
32	30		'ignore_text': [], # List of text to ignore when calculating the comparison checksum
33	31		# Custom notification content
34	32		'notification_urls': [], # List of URLs to add to the notification Queue (Usually AppRise)
35		-	'notification_title': default_notification_title,
36		-	'notification_body': default_notification_body,
37		-	'notification_format': default_notification_format,
38		-	'notification_use_default': True, # Use default for new
	33	+	'notification_title': None,
	34	+	'notification_body': None,
	35	+	'notification_format': default_notification_format_for_watch,
39	36		'notification_muted': False,
40	37		'css_filter': '',
41	38		'last_error': False,
		skipped 214 lines

■ ■ ■ ■ ■ ■

changedetectionio/notification.py

		skipped 13 lines
14	14		'current_snapshot': ''
15	15		}
16	16
	17	+	default_notification_format_for_watch = 'System default'
	18	+	default_notification_format = 'Text'
	19	+	default_notification_body = '{watch_url} had a change.\n---\n{diff}\n---\n'
	20	+	default_notification_title = 'ChangeDetection.io Notification - {watch_url}'
	21	+
17	22		valid_notification_formats = {
18	23		'Text': NotifyFormat.TEXT,
19	24		'Markdown': NotifyFormat.MARKDOWN,
20	25		'HTML': NotifyFormat.HTML,
	26	+	# Used only for editing a watch (not for global)
	27	+	default_notification_format_for_watch: default_notification_format_for_watch
21	28		}
22		-
23		-	default_notification_format = 'Text'
24		-	default_notification_body = '{watch_url} had a change.\n---\n{diff}\n---\n'
25		-	default_notification_title = 'ChangeDetection.io Notification - {watch_url}'
26	29
27	30		def process_notification(n_object, datastore):
28	31
		skipped 140 lines

changedetectionio/static/images/notice.svg

changedetectionio/static/images/play.svg

■ ■ ■ ■ ■ ■

changedetectionio/static/js/watch-settings.js

1		-	$(document).ready(function () {
2		-	function toggle_fetch_backend() {
	1	+	$(document).ready(function() {
	2	+	function toggle() {
3	3		if ($('input[name="fetch_backend"]:checked').val() == 'html_webdriver') {
4		-	if (playwright_enabled) {
	4	+	if(playwright_enabled) {
5	5		// playwright supports headers, so hide everything else
6	6		// See #664
7	7		$('#requests-override-options #request-method').hide();
		skipped 5 lines
13	13		// selenium/webdriver doesnt support anything afaik, hide it all
14	14		$('#requests-override-options').hide();
15	15		}
	16	+
	17	+
16	18		$('#webdriver-override-options').show();
	19	+
17	20		} else {
	21	+
18	22		$('#requests-override-options').show();
19	23		$('#requests-override-options *:hidden').show();
20	24		$('#webdriver-override-options').hide();
		skipped 1 lines
22	26		}
23	27
24	28		$('input[name="fetch_backend"]').click(function (e) {
25		-	toggle_fetch_backend();
	29	+	toggle();
26	30		});
27		-	toggle_fetch_backend();
28		-
29		-	function toggle_default_notifications() {
30		-	var n=$('#notification_urls, #notification_title, #notification_body, #notification_format');
31		-	if ($('#notification_use_default').is(':checked')) {
32		-	$('#notification-field-group').fadeOut();
33		-	$(n).each(function (e) {
34		-	$(this).attr('readonly', true);
35		-	});
36		-	} else {
37		-	$('#notification-field-group').show();
38		-	$(n).each(function (e) {
39		-	$(this).attr('readonly', false);
40		-	});
41		-	}
42		-	}
	31	+	toggle();
43	32
44		-	$('#notification_use_default').click(function (e) {
45		-	toggle_default_notifications();
	33	+	$('#notification-setting-reset-to-default').click(function (e) {
	34	+	$('#notification_title').val('');
	35	+	$('#notification_body').val('');
	36	+	$('#notification_format').val('System default');
	37	+	$('#notification_urls').val('');
	38	+	e.preventDefault();
46	39		});
47		-	toggle_default_notifications();
48	40		});
49	41

■ ■ ■ ■ ■ ■

changedetectionio/static/styles/styles.css

		skipped 565 lines
566	566		.checkbox-uuid > * {
567	567		vertical-align: middle; }
568	568
	569	+	.inline-warning {
	570	+	border: 1px solid #ff3300;
	571	+	padding: 0.5rem;
	572	+	border-radius: 5px;
	573	+	color: #ff3300; }
	574	+	.inline-warning > span {
	575	+	display: inline-block;
	576	+	vertical-align: middle; }
	577	+	.inline-warning img.inline-warning-icon {
	578	+	display: inline;
	579	+	height: 26px;
	580	+	vertical-align: middle; }
	581	+

■ ■ ■ ■ ■ ■

changedetectionio/static/styles/styles.scss

		skipped 786 lines
787	787		}
788	788		}
789	789
	790	+	.inline-warning {
	791	+	> span {
	792	+	display: inline-block;
	793	+	vertical-align: middle;
	794	+	}
	795	+
	796	+	img.inline-warning-icon {
	797	+	display: inline;
	798	+	height: 26px;
	799	+	vertical-align: middle;
	800	+	}
	801	+
	802	+	border: 1px solid #ff3300;
	803	+	padding: 0.5rem;
	804	+	border-radius: 5px;
	805	+	color: #ff3300;
	806	+	}

■ ■ ■ ■ ■ ■

changedetectionio/store.py

		skipped 536 lines
537	537		continue
538	538		return
539	539
540		-
541	540		def update_5(self):
542		-
543		-	from changedetectionio.notification import (
544		-	default_notification_body,
545		-	default_notification_format,
546		-	default_notification_title,
547		-	)
548		-
	541	+	# If the watch notification body, title look the same as the global one, unset it, so the watch defaults back to using the main settings
	542	+	# In other words - the watch notification_title and notification_body are not needed if they are the same as the default one
	543	+	current_system_body = self.data['settings']['application']['notification_body'].translate(str.maketrans('', '', "\r\n "))
	544	+	current_system_title = self.data['settings']['application']['notification_body'].translate(str.maketrans('', '', "\r\n "))
549	545		for uuid, watch in self.data['watching'].items():
550	546		try:
551		-	# If it's all the same to the system settings, then prefer system notification settings
552		-	# include \r\n -> \n incase they already hit submit and the browser put \r in
553		-	if watch.get('notification_body').replace('\r\n', '\n') == default_notification_body.replace('\r\n', '\n') and \
554		-	watch.get('notification_format') == default_notification_format and \
555		-	watch.get('notification_title').replace('\r\n', '\n') == default_notification_title.replace('\r\n', '\n') and \
556		-	watch.get('notification_urls') == self.__data['settings']['application']['notification_urls']:
557		-	watch['notification_use_default'] = True
558		-	else:
559		-	watch['notification_use_default'] = False
560		-	except:
	547	+	watch_body = watch.get('notification_body', '')
	548	+	if watch_body and watch_body.translate(str.maketrans('', '', "\r\n ")) == current_system_body:
	549	+	# Looks the same as the default one, so unset it
	550	+	watch['notification_body'] = None
	551	+
	552	+	watch_title = watch.get('notification_title', '')
	553	+	if watch_title and watch_title.translate(str.maketrans('', '', "\r\n ")) == current_system_title:
	554	+	# Looks the same as the default one, so unset it
	555	+	watch['notification_title'] = None
	556	+	except Exception as e:
561	557		continue
562	558		return
	559	+
	560	+

■ ■ ■ ■ ■ ■

changedetectionio/templates/_common_fields.jinja

1	1
2	2		{% from '_helpers.jinja' import render_field %}
3	3
4		-	{% macro render_common_settings_form(form, current_base_url, emailprefix) %}
	4	+	{% macro render_common_settings_form(form, emailprefix, settings_application) %}
5	5		<div class="pure-control-group">
6	6		{{ render_field(form.notification_urls, rows=5, placeholder="Examples:
7	7		Gitter - gitter://token/room
8	8		Office365 - o365://TenantID:AccountEmail/ClientID/ClientSecret/TargetEmail
9	9		AWS SNS - sns://AccessKeyID/AccessSecretKey/RegionName/+PhoneNo
10		-	SMTPS - mailtos://user:[email protected][email protected]", class="notification-urls")
	10	+	SMTPS - mailtos://user:[email protected][email protected]",
	11	+	class="notification-urls" )
11	12		}}
12	13		<div class="pure-form-message-inline">
13	14		<ul>
		skipped 12 lines
26	27		</div>
27	28		<div id="notification-customisation" class="pure-control-group">
28	29		<div class="pure-control-group">
29		-	{{ render_field(form.notification_title, class="m-d notification-title") }}
	30	+	{{ render_field(form.notification_title, class="m-d notification-title", placeholder=settings_application['notification_title']) }}
30	31		<span class="pure-form-message-inline">Title for all notifications</span>
31	32		</div>
32	33		<div class="pure-control-group">
33		-	{{ render_field(form.notification_body , rows=5, class="notification-body") }}
	34	+	{{ render_field(form.notification_body , rows=5, class="notification-body", placeholder=settings_application['notification_body']) }}
34	35		<span class="pure-form-message-inline">Body for all notifications</span>
35	36		</div>
36	37		<div class="pure-control-group">
37		-	{{ render_field(form.notification_format , rows=5, class="notification-format") }}
	38	+	<!-- unsure -->
	39	+	{{ render_field(form.notification_format , class="notification-format") }}
38	40		<span class="pure-form-message-inline">Format for all notifications</span>
39	41		</div>
40	42		<div class="pure-controls">
		skipped 53 lines
94	96		</table>
95	97		<br/>
96	98		URLs generated by changedetection.io (such as <code>{diff_url}</code>) require the <code>BASE_URL</code> environment variable set.<br/>
97		-	Your <code>BASE_URL</code> var is currently "{{current_base_url}}"
	99	+	Your <code>BASE_URL</code> var is currently "{{settings_application['current_base_url']}}"
98	100		</span>
99	101		</div>
100	102		</div>
		skipped 2 lines

■ ■ ■ ■ ■ ■

changedetectionio/templates/edit.html

		skipped 136 lines
137	137		<div class="tab-pane-inner" id="notifications">
138	138		<fieldset>
139	139		<div class="pure-control-group inline-radio">
140		-	{{ render_checkbox_field(form.notification_use_default) }}
	140	+	{{ render_checkbox_field(form.notification_muted) }}
141	141		</div>
142	142		<div class="field-group" id="notification-field-group">
143		-	{{ render_common_settings_form(form, current_base_url, emailprefix) }}
	143	+	{% if has_default_notification_urls %}
	144	+	<div class="inline-warning">
	145	+	<img class="inline-warning-icon" src="{{url_for('static_content', group='images', filename='notice.svg')}}" alt="Look out!" title="Lookout!"/>
	146	+	There are <a href="{{ url_for('settings_page')}}#notifications">system-wide notification URLs enabled</a>, this form will override notification settings for this watch only &dash; an empty Notification URL list here will still send notifications.
	147	+	</div>
	148	+	{% endif %}
	149	+	<a href="#notifications" id="notification-setting-reset-to-default" class="pure-button button-xsmall" style="right: 20px; top: 20px; position: absolute; background-color: #5f42dd; border-radius: 4px; font-size: 70%; color: #fff">Use system defaults</a>
	150	+
	151	+	{{ render_common_settings_form(form, emailprefix, settings_application) }}
144	152		</div>
145	153		</fieldset>
146	154		</div>
		skipped 179 lines

■ ■ ■ ■ ■ ■

changedetectionio/templates/settings.html

		skipped 59 lines
60	60		{{ render_field(form.application.form.base_url, placeholder="http://yoursite.com:5000/",
61	61		class="m-d") }}
62	62		<span class="pure-form-message-inline">
63		-	Base URL used for the {base_url} token in notifications and RSS links.<br/>Default value is the ENV var 'BASE_URL' (Currently "{{current_base_url}}"),
	63	+	Base URL used for the <code>{base_url}</code> token in notifications and RSS links.<br/>Default value is the ENV var 'BASE_URL' (Currently "{{settings_application['current_base_url']}}"),
64	64		<a href="https://github.com/dgtlmoon/changedetection.io/wiki/Configurable-BASE_URL-setting">read more here</a>.
65	65		</span>
66	66		</div>
		skipped 20 lines
87	87		<div class="tab-pane-inner" id="notifications">
88	88		<fieldset>
89	89		<div class="field-group">
90		-	{{ render_common_settings_form(form.application.form, current_base_url, emailprefix) }}
	90	+	{{ render_common_settings_form(form.application.form, emailprefix, settings_application) }}
91	91		</div>
92	92		</fieldset>
93	93		</div>
		skipped 92 lines

■ ■ ■ ■ ■ ■

changedetectionio/templates/watch-overview.html

		skipped 29 lines
30	30		<div id="checkbox-operations">
31	31		<button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="pause">Pause</button>
32	32		<button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="unpause">UnPause</button>
	33	+	<button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="mute">Mute</button>
	34	+	<button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="unmute">UnMute</button>
	35	+	<button class="pure-button button-secondary button-xsmall" style="font-size: 70%" name="op" value="notification-default">Use default notification</button>
33	36		<button class="pure-button button-secondary button-xsmall" style="background: #dd4242; font-size: 70%" name="op" value="delete">Delete</button>
34	37		</div>
35	38		<div>
		skipped 40 lines
76	79		{% if watch.uuid in queued_uuids %}queued{% endif %}">
77	80		<td class="inline checkbox-uuid" ><input name="uuids" type="checkbox" value="{{ watch.uuid}} "/> <span>{{ loop.index }}</span></td>
78	81		<td class="inline watch-controls">
79		-	<a class="state-{{'on' if watch.paused }}" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause checks" title="Pause checks"/></a>
	82	+	{% if not watch.paused %}
	83	+	<a class="state-off" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='pause.svg')}}" alt="Pause checks" title="Pause checks"/></a>
	84	+	{% else %}
	85	+	<a class="state-on" href="{{url_for('index', op='pause', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='play.svg')}}" alt="UnPause checks" title="UnPause checks"/></a>
	86	+	{% endif %}
80	87		<a class="state-{{'on' if watch.notification_muted}}" href="{{url_for('index', op='mute', uuid=watch.uuid, tag=active_tag)}}"><img src="{{url_for('static_content', group='images', filename='bell-off.svg')}}" alt="Mute notifications" title="Mute notifications"/></a>
81	88		</td>
82	89		<td class="title-col inline">{{watch.title if watch.title is not none and watch.title\|length > 0 else watch.url}}
		skipped 61 lines

■ ■ ■ ■ ■ ■

changedetectionio/tests/test_notification.py

		skipped 3 lines
4	4		from flask import url_for
5	5		from . util import set_original_response, set_modified_response, set_more_modified_response, live_server_setup
6	6		import logging
7		-	from changedetectionio.notification import default_notification_body, default_notification_title
	7	+
	8	+	from changedetectionio.notification import (
	9	+	default_notification_body,
	10	+	default_notification_format,
	11	+	default_notification_title,
	12	+	valid_notification_formats,
	13	+	)
8	14
9	15		def test_setup(live_server):
10	16		live_server_setup(live_server)
		skipped 9 lines
20	26
21	27		# Re 360 - new install should have defaults set
22	28		res = client.get(url_for("settings_page"))
	29	+	notification_url = url_for('test_notification_endpoint', _external=True).replace('http', 'json')
	30	+
23	31		assert default_notification_body.encode() in res.data
24	32		assert default_notification_title.encode() in res.data
25	33
	34	+	#####################
	35	+	# Set this up for when we remove the notification from the watch, it should fallback with these details
	36	+	res = client.post(
	37	+	url_for("settings_page"),
	38	+	data={"application-notification_urls": notification_url,
	39	+	"application-notification_title": "fallback-title "+default_notification_title,
	40	+	"application-notification_body": "fallback-body "+default_notification_body,
	41	+	"application-notification_format": default_notification_format,
	42	+	"requests-time_between_check-minutes": 180,
	43	+	'application-fetch_backend': "html_requests"},
	44	+	follow_redirects=True
	45	+	)
	46	+
	47	+	assert b"Settings updated." in res.data
	48	+
26	49		# When test mode is in BASE_URL env mode, we should see this already configured
27	50		env_base_url = os.getenv('BASE_URL', '').strip()
28	51		if len(env_base_url):
		skipped 18 lines
47	70
48	71		# Goto the edit page, add our ignore text
49	72		# Add our URL to the import page
50		-	url = url_for('test_notification_endpoint', _external=True)
51		-	notification_url = url.replace('http', 'json')
52	73
53	74		print (">>>> Notification URL: "+notification_url)
54	75
		skipped 16 lines
71	92		"url": test_url,
72	93		"tag": "my tag",
73	94		"title": "my title",
74		-	# No 'notification_use_default' here, so it's effectively False/off
75	95		"headers": "",
76	96		"fetch_backend": "html_requests"})
77	97
		skipped 81 lines
159	179		# be sure we see it in the output log
160	180		assert b'New ChangeDetection.io Notification - ' + test_url.encode('utf-8') in res.data
161	181
	182	+	set_original_response()
	183	+	res = client.post(
	184	+	url_for("edit_page", uuid="first"),
	185	+	data={
	186	+	"url": test_url,
	187	+	"tag": "my tag",
	188	+	"title": "my title",
	189	+	"notification_urls": '',
	190	+	"notification_title": '',
	191	+	"notification_body": '',
	192	+	"notification_format": default_notification_format,
	193	+	"fetch_backend": "html_requests"},
	194	+	follow_redirects=True
	195	+	)
	196	+	assert b"Updated watch." in res.data
	197	+
	198	+	time.sleep(2)
	199	+
	200	+	# Verify what was sent as a notification, this file should exist
	201	+	with open("test-datastore/notification.txt", "r") as f:
	202	+	notification_submission = f.read()
	203	+	assert "fallback-title" in notification_submission
	204	+	assert "fallback-body" in notification_submission
	205	+
162	206		# cleanup for the next
163	207		client.get(
164	208		url_for("form_delete", uuid="all"),
		skipped 16 lines
181	225		assert b"Watch added" in res.data
182	226
183	227		# Re #360 some validation
184		-	res = client.post(
185		-	url_for("edit_page", uuid="first"),
186		-	data={"notification_urls": 'json://localhost/foobar',
187		-	"notification_title": "",
188		-	"notification_body": "",
189		-	"notification_format": "Text",
190		-	"url": test_url,
191		-	"tag": "my tag",
192		-	"title": "my title",
193		-	"headers": "",
194		-	"fetch_backend": "html_requests"},
195		-	follow_redirects=True
196		-	)
197		-	assert b"Notification Body and Title is required when a Notification URL is used" in res.data
	228	+	# res = client.post(
	229	+	# url_for("edit_page", uuid="first"),
	230	+	# data={"notification_urls": 'json://localhost/foobar',
	231	+	# "notification_title": "",
	232	+	# "notification_body": "",
	233	+	# "notification_format": "Text",
	234	+	# "url": test_url,
	235	+	# "tag": "my tag",
	236	+	# "title": "my title",
	237	+	# "headers": "",
	238	+	# "fetch_backend": "html_requests"},
	239	+	# follow_redirects=True
	240	+	# )
	241	+	# assert b"Notification Body and Title is required when a Notification URL is used" in res.data
198	242
199	243		# Now adding a wrong token should give us an error
200	244		res = client.post(
		skipped 16 lines
217	261		follow_redirects=True
218	262		)
219	263
220		-	# Check that the default VS watch specific notification is hit
221		-	def test_check_notification_use_default(client, live_server):
222		-	set_original_response()
223		-	notification_url = url_for('test_notification_endpoint', _external=True).replace('http', 'json')
224		-	test_url = url_for('test_endpoint', _external=True)
225	264
226		-	res = client.post(
227		-	url_for("form_quick_watch_add"),
228		-	data={"url": test_url, "tag": ''},
229		-	follow_redirects=True
230		-	)
231		-	assert b"Watch added" in res.data
232	265
233		-	## Setup the local one and enable it
234		-	res = client.post(
235		-	url_for("edit_page", uuid="first"),
236		-	data={"notification_urls": notification_url,
237		-	"notification_title": "watch-notification",
238		-	"notification_body": "watch-body",
239		-	'notification_use_default': "True",
240		-	"notification_format": "Text",
241		-	"url": test_url,
242		-	"tag": "my tag",
243		-	"title": "my title",
244		-	"headers": "",
245		-	"fetch_backend": "html_requests"},
246		-	follow_redirects=True
247		-	)
248		-
249		-	res = client.post(
250		-	url_for("settings_page"),
251		-	data={"application-notification_title": "global-notifications-title",
252		-	"application-notification_body": "global-notifications-body\n",
253		-	"application-notification_format": "Text",
254		-	"application-notification_urls": notification_url,
255		-	"requests-time_between_check-minutes": 180,
256		-	"fetch_backend": "html_requests"
257		-	},
258		-	follow_redirects=True
259		-	)
260		-
261		-	# A change should by default trigger a notification of the global-notifications
262		-	time.sleep(1)
263		-	set_modified_response()
264		-	client.get(url_for("form_watch_checknow"), follow_redirects=True)
265		-	time.sleep(2)
266		-	with open("test-datastore/notification.txt", "r") as f:
267		-	assert 'global-notifications-title' in f.read()
268		-
269		-	## Setup the local one and enable it
270		-	res = client.post(
271		-	url_for("edit_page", uuid="first"),
272		-	data={"notification_urls": notification_url,
273		-	"notification_title": "watch-notification",
274		-	"notification_body": "watch-body",
275		-	# No 'notification_use_default' here, so it's effectively False/off = "dont use default, use this one"
276		-	"notification_format": "Text",
277		-	"url": test_url,
278		-	"tag": "my tag",
279		-	"title": "my title",
280		-	"headers": "",
281		-	"fetch_backend": "html_requests"},
282		-	follow_redirects=True
283		-	)
284		-	set_original_response()
285		-
286		-	client.get(url_for("form_watch_checknow"), follow_redirects=True)
287		-	time.sleep(2)
288		-	assert os.path.isfile("test-datastore/notification.txt")
289		-	with open("test-datastore/notification.txt", "r") as f:
290		-	assert 'watch-notification' in f.read()
291		-
292		-
293		-	# cleanup for the next
294		-	client.get(
295		-	url_for("form_delete", uuid="all"),
296		-	follow_redirects=True
297		-	)

■ ■ ■ ■ ■ ■

changedetectionio/update_worker.py

		skipped 10 lines
11	11		# Requests for checking on a single site(watch) from a queue of watches
12	12		# (another process inserts watches into the queue that are time-ready for checking)
13	13
	14	+	import logging
	15	+	import sys
14	16
15	17		class update_worker(threading.Thread):
16	18		current_uuid = None
17	19
18	20		def __init__(self, q, notification_q, app, datastore, args, *kwargs):
	21	+	logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
19	22		self.q = q
20	23		self.app = app
21	24		self.notification_q = notification_q
		skipped 4 lines
26	29
27	30		from changedetectionio import diff
28	31
	32	+	from changedetectionio.notification import (
	33	+	default_notification_format_for_watch
	34	+	)
	35	+
29	36		n_object = {}
30	37		watch = self.datastore.data['watching'].get(watch_uuid, False)
31	38		if not watch:
		skipped 8 lines
40	47		"History index had 2 or more, but only 1 date loaded, timestamps were not unique? maybe two of the same timestamps got written, needs more delay?"
41	48		)
42	49
43		-	# Did it have any notification alerts to hit?
44		-	if not watch.get('notification_use_default') and len(watch['notification_urls']):
45		-	print(">>> Notifications queued for UUID from watch {}".format(watch_uuid))
46		-	n_object['notification_urls'] = watch['notification_urls']
47		-	n_object['notification_title'] = watch['notification_title']
48		-	n_object['notification_body'] = watch['notification_body']
49		-	n_object['notification_format'] = watch['notification_format']
	50	+	n_object['notification_urls'] = watch['notification_urls'] if len(watch['notification_urls']) else \
	51	+	self.datastore.data['settings']['application']['notification_urls']
	52	+
	53	+	n_object['notification_title'] = watch['notification_title'] if watch['notification_title'] else \
	54	+	self.datastore.data['settings']['application']['notification_title']
	55	+
	56	+	n_object['notification_body'] = watch['notification_body'] if watch['notification_body'] else \
	57	+	self.datastore.data['settings']['application']['notification_body']
	58	+
	59	+	n_object['notification_format'] = watch['notification_format'] if watch['notification_format'] != default_notification_format_for_watch else \
	60	+	self.datastore.data['settings']['application']['notification_format']
50	61
51		-	# No? maybe theres a global setting, queue them all
52		-	elif watch.get('notification_use_default') and len(self.datastore.data['settings']['application']['notification_urls']):
53		-	print(">>> Watch notification URLs were empty, using GLOBAL notifications for UUID: {}".format(watch_uuid))
54		-	n_object['notification_urls'] = self.datastore.data['settings']['application']['notification_urls']
55		-	n_object['notification_title'] = self.datastore.data['settings']['application']['notification_title']
56		-	n_object['notification_body'] = self.datastore.data['settings']['application']['notification_body']
57		-	n_object['notification_format'] = self.datastore.data['settings']['application']['notification_format']
58		-	else:
59		-	print(">>> NO notifications queued, watch and global notification URLs were empty.")
60	62
61	63		# Only prepare to notify if the rules above matched
62		-	if 'notification_urls' in n_object:
	64	+	if 'notification_urls' in n_object and n_object['notification_urls']:
63	65		# HTML needs linebreak, but MarkDown and Text can use a linefeed
64	66		if n_object['notification_format'] == 'HTML':
65	67		line_feed_sep = "</br>"
66	68		else:
67	69		line_feed_sep = "\n"
68	70
69		-	snapshot_contents = ''
70	71		with open(watch_history[dates[-1]], 'rb') as f:
71	72		snapshot_contents = f.read()
72	73
		skipped 4 lines
77	78		'diff': diff.render_diff(watch_history[dates[-2]], watch_history[dates[-1]], line_feed_sep=line_feed_sep),
78	79		'diff_full': diff.render_diff(watch_history[dates[-2]], watch_history[dates[-1]], True, line_feed_sep=line_feed_sep)
79	80		})
80		-
	81	+	logging.info (">> SENDING NOTIFICATION")
81	82		self.notification_q.put(n_object)
	83	+	else:
	84	+	logging.info (">> NO Notification sent, notification_url was empty in both watch and system")
82	85
83	86		def send_filter_failure_notification(self, watch_uuid):
84	87
		skipped 97 lines
182	185		process_changedetection_results = False
183	186
184	187		except FilterNotFoundInResponse as e:
	188	+	if not self.datastore.data['watching'].get(uuid):
	189	+	continue
	190	+
185	191		err_text = "Warning, filter '{}' not found".format(str(e))
186	192		self.datastore.update_watch(uuid=uuid, update_obj={'last_error': err_text,
187	193		# So that we get a trigger when the content is added again
		skipped 109 lines

■ ■ ■ ■ ■ ■

docker-compose.yml

		skipped 29 lines
30	30		#
31	31		# https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch-option-proxy
32	32		#
33		-	# Plain requsts - proxy support example.
	33	+	# Plain requests - proxy support example.
34	34		# - HTTP_PROXY=socks5h://10.10.1.10:1080
35	35		# - HTTPS_PROXY=socks5h://10.10.1.10:1080
36	36		#
		skipped 55 lines

Merge branch 'master' into fetchers-abstract