DS3231 RTC As NTP Client Replacement for ESP32

24 Views, 1 Favorites, 0 Comments

DS3231 RTC As NTP Client Replacement for ESP32

This Instructable shows you how to use the DS3231 RTC (Real Time Clock) and the RTC_NTP_replacement Arduino library to replace the ESP32's NTP client for completely off-line use.

Even if you opt to use NTP, the RTC replacement lets your sketch startup more reliably and faster as it does not have to wait for the WiFi connection and for the NTP server to respond. The sketch will also run even if the WiFi or NTP server fails to connect.

This RTC_NTP_replacement library is for the ESP32 and DS3231 RTC and has the following features:-

– Runs completely off-line, using the RTC (DS3231) to synchronize the ESP32's system clock to within a few milliseconds.

– Uses the standard ESP32 time methods to get times and dates. The RTC is not accessed from user code.

– Provides 'smooth' synchronization of the ESP32's system clock to the RTC's time for consistent time stamps.

– Provides locally served web pages to set the time and the time zone. The web pages are formatted for easy display on a mobile phone.

– Has a time zone parser to clean up the user inputted time zone to a valid one that the ESP32 can use.

– Can optionally to use an NTP server to synchronize the RTC's date and time which then synchronizes the system time.

– Can optionally to use AsyncWebServer instead of ESP32's built in web server

In this library, the RTC's time and date is the master UTC time. Having set the RTC's time and the ESP32's time zone from the built-in web page's, the ESP32's unix time is keep in sync with the RTC's time such that the unix time never goes backwards. If the ESP32's system time is ahead of the RTC, then the system clock is halted until the RTC catches up. If the ESP32's system time is behind the RTC's, the system time is pushed forward two seconds per second until it catches up with the RTC.

The system runs completely off-line, the web pages to set the RTC's data and time and the ESP32's time zone are served locally, using the ESP32's basic web server, via an Access Point created by the ESP32, see the RTC_TZ.ino example. While the DS3231 RTC is very accurate (+/- 0.3sec / day), it will drift over time, up to 10 secs a month. The web page can be used to adjust the RTC time as necessary. If the RTC adjustment is greater then 5 mins, then the ESP32 system time is updated immediately as well. Otherwise just the RTC time is updated and the ESP32 system time is 'smoothly' synchronised to match.

The RTC runs on UTC time zone, as does the ESP32 system clock. Local time and dates are produced by applying the time zone set in the ESP32 environmental variable. The ESP32's time zone is set via a Set Time Zone web page that allows the user to enter the Posix timezone string for their timezone. The library then parses and cleans up this input to give both a valid timezone string and a description of what the string means, for the user to accept. A default time zone can be set in code, otherwise the GMT0 is used.

Changing the ESP32's time zone does NOT change the RTC's time or the ESP32's system time. Also daylight saving changes do not change the RTC's time or the ESP32's system time, so you can use the ESP32's unix time as a consistent timestamp and convert it to local time for display. The RTC synchronization ensures the ESP32's unix timestamp does not go backwards.

Even if you normally have internet available, and opt use the NTP to synchronize the RTC, this library provides a faster more reliable startup as it sets the system time from the RTC in setup() and does not have to wait for the WiFi connection to be made and for the NTP server to respond before your code returns reliable timestamps. The code runs even if the WiFi connection fails.

Note: The ESP32's gettimeofday() method returns system's current seconds and microseconds. The RTC only returns times to the second. However this library uses the 1 Hz square wave output from the RTC to synchronize the system's microseconds also, to within a +/- 5 milliseconds. This synchronisation is tolerant of loop() processing delays caused by other code.

When the RTC is synchronized from an NTP server, the synchronisation is delayed until the next whole second to improve the millisecond accuracy. However this delay and hence the millisecond accuracy of the RTC with respect to the NTP time, is affected by other loop() processing delays, so you need to keep the loop responsive and/or call the handleNTP() method at multiple places through out your code. See Simple Multitasking Arduino for the details and how to add a looptimer to check your loop's responsiveness.

Supplies

1 x Dfrobot ESP32 FireBeetle 2 ESP32-E (~US$8.90) or other ESP32 board

1 x Dfrobot Fermion: DS3231 Precise RTC (~US$8.90) or other DS3231 RTC

1 x ESP32 Sketch Data Upload https://github.com/me-no-dev/arduino-esp32fs-plugin

ESP32 Arduino Board Support V3.0.7 (other versions not tested)

RTC_NTP_replacement library .zip file. Add to the Arduino IDE either by Sketch → Include Library → Add .zip library.. or via download via the Arduino library manager.

Schematic

The schematic is trivial. Line up the DFROBOT's DS3231 module with the 3V3, GND, SCL, SDA, A4 (GPIO15) pins and wire directly across.

The GPIO15 (A4) input is used to connect to the 1Hz output of the DS3231 to trigger an interrupt each time a second ticks over.

Loading the Web Pages Used by the Library

To use this library you need to install setup web pages on the ESP32, using the Sketch Data Upload plugin.

See the instructions at https://github.com/me-no-dev/arduino-esp32fs-plugin

Download https://github.com/me-no-dev/arduino-esp32fs-plugin/releases/download/1.1/ESP32FS-1.1.zip

In your Arduino sketchbook directory, create tools directory if it doesn't exist yet.

Unpack the tool into tools directory (the path will look like C:\Users\<username>\Arduino\tools\ESP32FS\tool\esp32fs.jar).

Restart Arduino IDE.

On the OS X create the tools directory in ~/Documents/Arduino/ and unpack the files there

Select Tools > ESP32 Sketch Data Upload menu item. Upload as LittleFS. That will upload the files in the current sketch's data subdirectory to the ESP32

The same set of web pages is used for all four examples.

Setting the Time and Time Zone

After loading the RTC_TZ.ino example and its data, you can connect your mobile to the RTCwithTZ access point (password 12345678) and open the http://192.168.1.1/index.html web page to set the time and time zone.

NOTE: Make sure to use http://192.168.1.1/index.html, not https.

When first powered up, with the battery installed, a default date and time of 2020/10/10 10:10:00:10 is set in the RTC and the index.html web page shows that the RTC date and time has not been set.

You can then open the Set Time web page to set the current local time based on the current time zone setting.

A default time zone can be set in your sketch. For example at the top of RTC_TZ.ino there is

const char *get_DefaultTZ() { // magic name picked up by TZ_Support.cpp

// TZ_Australia_Sydney

static char _default_tz_[50] = "AEST-10AEDT,M10.1.0,M4.1.0/3";

return _default_tz_;

}

Which sets Sydney, Australia's time zone as the default. You can change this to your time zone.

Updating the Time Zone by Setting the POSIX String

If you want to change your default time zone, you can enter the appropriate POSIX string to set the time zone and its daylight saving rules (if any)

See https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv for a list of time zones and their posix_tz strings that can be copied and pasted into the get_DefaultTZ or into the Set Time Zone web page.

See Explanation of TZ strings (local copy here) for how this string is constructed. When you click Set TZ Format the input string is parsed and cleaned up and if the cleaned up string is different from what you entered then it is re-displayed for your approval or further corrections.

Once the input string equals the parsed/cleaned up string, that TZ is accepted as valid and the you are returned to the Set Time page to set the current local time in this new time zone.

The parser is a simple one and can be easily confused by invalid formats. At a minimum you need to specify the number of hours:mins to add to your timezone to get back to GMT. e.g.

-10 for GMT+10

5:30 for GMT-0530 The : in the POSIX string is important.

Next if your timezone has daylight saving then you need to add a comma and specify the start of dst month . week . dayOfWeek and optionally /hr which defaults to 2am if missing. This is followed by a , and when the details of when dst ends. The parser/cleanup will fill in defaults for the week (5 == last week in the month) and dayOfWeek (0 == Sunday) if missing, and if the end of dst section is missing it will default to 6 months later. The default dst shift is 1hr.

E.g. entering

8,10

will be parsed and cleaned up to

<-08>8<-07>7,M10.1.0/2,M4.1.0/2

There is popup help for the time zone format <Click here for Format Help>

Finally there is Reset Default Time Zone button to restore the time zone setting that was programmed into the your sketch via the get_DefaultTZ() method.

Using the Library

A Basic Off-Line Sketch

A basic sketch which runs completely off-line using ESP32's Access Point. See RTC_TZ.ino for the complete code

#include "RTC_NTP_replacement.h"

const int SQUARE_WAVE_PIN = 15; // the pin the RTC 1Hz output is connected to

// set a compiled default TZ here. Can be overrided/edited later by web page.

// see https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv for tz strings

const char *get_DefaultTZ() {

static char _default_tz_[50] = "AEST-10AEDT,M10.1.0,M4.1.0/3"; // TZ_Australia_Sydney

return _default_tz_;

}

void setup() {

initTZsupport(); // load current tz from file

initRTC(SDA, SCL, SQUARE_WAVE_PIN ); // start RTC and interrupt monitor

setupAP(); // start ESP32 Access Point

}

void loop() {

syncFromRTC(); // monitor interrupt and keep system time in sync with RTC

handleWebServer(); // handle ESP32 web server. This starts web server on first call

}

A Basic Sketch with NTP updates if Available

A basic sketch with NTP updates. See RTC_NTP example for the complete code

#include "RTC_NTP_replacement.h"

#include "WiFi.h"

const int SQUARE_WAVE_PIN = 15; // the pin the RTC 1Hz output is connected to

//// WiFi credentials - replace with your own

const char* ssid = "SSID";

const char* password = "password";

IPAddress staticIP(10, 1, 1, 253); // static Ip

// Set a compiled default TZ here. Can be overrided/edited later by web page.

// See https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv for tz strings

const char *get_DefaultTZ() {

static char _default_tz_[50] = "AEST-10AEDT,M10.1.0,M4.1.0/3"; // TZ_Australia_Sydney

return _default_tz_;

}

// . . . connectToWiFi() method here

void setup() {

initNTP(); // NOTE!!! call this BEFORE calling initTZsupport() and initRTC()

// because configTime( ) has 0 for the hr and min offset

initTZsupport(); // starts FS by calling initializeFS()

initRTC(SDA, SCL, SQUARE_WAVE_PIN );

}

void loop() {

connectToWiFi(); // try to connect,

// call this before calling handleWebServer()

// as WiFi.begin needs to be called before starting ESP32 webserver

handleWebServer(); //starts web server on first call

syncFromRTC(); // keep system time in sync with RTC

handleNTP(); // update RTC from NTP response when it arrives

}

Note that the connectToWiFi() method is called in the loop() and not in startup() so that the sketch does not block in startup() waiting for a WiFi connection that may not succeed.

A Basic Sketch using AsyncWebServer

If your other code uses the AsyncWebServer then it should also be used to serve the Set Time and Set Time Zone web pages.

See RTC_NTP_Async example for the complete code

#include "RTC_NTP_replacement.h"

#include "WiFi.h"

#include "ESPAsyncWebServer.h"

const int SQUARE_WAVE_PIN = 15; // the pin the RTC 1Hz output is connected to

//// WiFi credentials - replace with your own

const char* ssid = "SSID";

const char* password = "password";

IPAddress staticIP(10, 1, 1, 253); // static IP

// set a compiled default TZ here. Can be overrided/edited later by web page.

// see https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv for tz strings

const char *get_DefaultTZ() {

static char _default_tz_[50] = "AEST-10AEDT,M10.1.0,M4.1.0/3"; // TZ_Australia_Sydney

return _default_tz_;

}

// . . . connectToWiFi() method here

void setup() {

initNTP(); // NOTE!!! call this BEFORE calling initTZsupport() and initRTC()

// because configTime( ) has 0 for the hr and min offset

initTZsupport(); // starts FS by calling initializeFS()

initRTC(SDA, SCL, SQUARE_WAVE_PIN );

startAsyncWebServer(); // can start AsyncWebServer before calling WiFi.begin()

}

void loop() {

connectToWiFi(); // try to connect

syncFromRTC(); // keep system time in sync with RTC

handleNTP();

}

// . . . AsyncWebServer definition here

// uses RTC_webVariableProcessor to handle RTC page processing

Code Description

ESP32 System Time Synchronization

The user code does not access the RTC timestamps directly. Rather the RTC is used to synchronize the ESP32's system time, via settimeofday(). Both the RTC and the ESP32 system time use the UTC (GMT0) timezone. The user's sketch sets the timezone for display. That time zone can be also be set via a built-in web page.

The 1Hz output of the RTC is used to synchronize the ESP32's system time to within +/- 5ms. When the RTC's second register is written the 1Hz output goes high 500ms later and thereafter the 1Hz output goes low each time the RTC's second ticks over. The RTC_support.cpp file, the squareWaveInterrupt() method is attached to the input pin the 1Hz is connected to.

static void IRAM_ATTR squareWaveInterrupt() {

if (interruptTriggered_v) {

return; // still waiting to process last interrupt

}

interruptMicros_v = micros();

interruptTriggered_v = true;

}

That method captures the micros() when the interrupt occurred. Some time later when the loop() calls syncFromRTC() , the current micros() and the system time (seconds and us) is read via gettimeofday() together with the current RTC unix timestamp. If the system time is in sync with the RTC then the system time seconds will match the RTC timestamp AND the elapsed microseconds since the interrupt will match the timeofday microseconds. If there is more than +/- 5000 us (5ms) difference, then if the system time is behind, it will be updated with the current RTC timestamp and the elapsed microseconds. If the system time is ahead then the system time's seconds are frozen until the RTC catches up. This ensures the system's unix timestamps, which is in seconds, never goes backwards.

If there is a gross change in the RTC time, due, for example, to a change in time zone, and the difference exceeds +/- 5mins, then the ESP32 system time is just updated immediately.

Web Page Processing

The AsyncWebServer has a processing method that replaces web page %...% variables with values before serving the web page. The ESP32 web server does not have this facility. To overcome that, a simplified replacement method, String processVars(const String & input), was written (by Claude Code AI) and used to process a small header string that was then send before the .tail of the html file. Having identified a variable name, processVars calls RTC_webVariableProcessor(varName) to do the actual replacement. This varname replacement method can be reused by the AsyncWebServer processor, if the web pages are being served by it. See the RTC_NTP_Async example.

Claude Code AI was used to pre-process the Async web pages to move the %...% variables to the top of the file and assign them to local javascript variables that the rest of the web page could access.

The Claude Code command used was

For the three files settzRTC.head index.head setRTCtime.head

Move the %..% parameters to variables at the top of the .html files.

Then create a new file containing the contents of each as a separate String

Use \n\ to continue the lines and escape the " with \"

For example

String head1 = "\

<!DOCTYPE html>\n\

<html>\n\

etc

The only other change needes was to remove the %% escapes in the Async web pages. In the data sub-directory, there are two sets of web files. The .html files are for use by AsyncWebServer and the .tail files are for use by the ESP32 web server to send after sending the processed header string.

The ESP32 web server is also missing a redirect feature. This small method supply it.

void redirect(const char *url) {

server.sendHeader("Location", url);

server.send(307);

}

Conclusion

This instructable covered how to use the RTC_NTP_replacement library to provide a complete off-line replacement for an NTP client. Even if you opt to use NTP, the RTC replacement lets your sketch startup more reliably and faster as it does not have to wait for the WiFi connection and for the NTP server to respond. The sketch will also run even if the WiFi or NTP server fails to connect.