From fb31a3224fd534923c82b9710437a49a6961eb2d Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Mon, 9 Dec 2024 16:28:19 -0500 Subject: [PATCH] Add zone config troubleshooting guide v1 Fixes DOC-9210 Summary of changes: - Add a new page, 'Troubleshoot Replication Zones', to _The ZoneConfigonomicon (tm)_ - Update the 'Replication controls' page with more detailed info re: zone config inheritance hierarchy and behavior - Update the made-up examples in the 'Critical nodes endpoint' docs to point back to 'Troubleshoot Replication Zones' - Fix incorrect statements on the `ALTER RANGE` page since they're needed to map from range IDs returned by the critical nodes endpoint (mentioned in 'Troubleshoot Replication Zones') to actual schema objects - Add moar links (tm) from various zone config-related pages to the new troubleshooting guide and amongst themselves - Add a note to various zone config-related docs saying "most users should not do manual zone config changes, see Multi-region SQL and Zone Config Extensions instead" --- ...zone-configs-overwritten-during-restore.md | 1 + .../see-zone-config-troubleshooting-guide.md | 1 + .../v24.3/sidebar-data/troubleshooting.json | 6 + .../zone-configs/avoid-manual-zone-configs.md | 3 + .../v24.3/zone-config-inheritance-diagram.png | Bin 0 -> 85418 bytes src/current/v24.3/alter-database.md | 19 ++ src/current/v24.3/alter-index.md | 8 +- src/current/v24.3/alter-partition.md | 10 + src/current/v24.3/alter-range.md | 16 +- src/current/v24.3/alter-table.md | 4 +- src/current/v24.3/backup.md | 3 +- src/current/v24.3/cluster-api.md | 2 +- .../v24.3/cluster-setup-troubleshooting.md | 24 +- src/current/v24.3/common-errors.md | 3 +- .../v24.3/configure-replication-zones.md | 261 +++++++++++++++++- ...emo-low-latency-multi-region-deployment.md | 4 +- src/current/v24.3/local-testing.md | 4 +- .../v24.3/migrate-to-multiregion-sql.md | 1 + src/current/v24.3/monitoring-and-alerting.md | 20 +- src/current/v24.3/multiregion-overview.md | 1 + src/current/v24.3/node-shutdown.md | 2 +- src/current/v24.3/partitioning.md | 8 +- .../v24.3/query-replication-reports.md | 1 + src/current/v24.3/restore.md | 2 +- src/current/v24.3/show-backup.md | 2 +- src/current/v24.3/show-create.md | 2 +- src/current/v24.3/show-zone-configurations.md | 3 +- ...take-and-restore-locality-aware-backups.md | 2 +- .../v24.3/troubleshoot-replication-zones.md | 231 ++++++++++++++++ src/current/v24.3/troubleshooting-overview.md | 4 +- src/current/v24.3/ui-debug-pages.md | 2 +- src/current/v24.3/zone-config-extensions.md | 1 + 32 files changed, 599 insertions(+), 52 deletions(-) create mode 100644 src/current/_includes/v24.3/backups/zone-configs-overwritten-during-restore.md create mode 100644 src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md create mode 100644 src/current/_includes/v24.3/zone-configs/avoid-manual-zone-configs.md create mode 100644 src/current/images/v24.3/zone-config-inheritance-diagram.png create mode 100644 src/current/v24.3/troubleshoot-replication-zones.md diff --git a/src/current/_includes/v24.3/backups/zone-configs-overwritten-during-restore.md b/src/current/_includes/v24.3/backups/zone-configs-overwritten-during-restore.md new file mode 100644 index 00000000000..f17be399b3d --- /dev/null +++ b/src/current/_includes/v24.3/backups/zone-configs-overwritten-during-restore.md @@ -0,0 +1 @@ +[Zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) present on the destination cluster prior to a restore will be **overwritten** during a [cluster restore]({% link {{ page.version.version }}/restore.md %}#full-cluster) with the zone configurations from the [backed up cluster]({% link {{ page.version.version }}/backup.md %}#back-up-a-cluster). If there were no customized zone configurations on the cluster when the backup was taken, then after the restore the destination cluster will use the zone configuration from the [`RANGE DEFAULT` configuration]({% link {{ page.version.version }}/configure-replication-zones.md %}#view-the-default-replication-zone). diff --git a/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md b/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md new file mode 100644 index 00000000000..a3773ced626 --- /dev/null +++ b/src/current/_includes/v24.3/see-zone-config-troubleshooting-guide.md @@ -0,0 +1 @@ +For instructions showing how to troubleshoot replication zones, see [Troubleshoot Replication Zone Configurations]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}). diff --git a/src/current/_includes/v24.3/sidebar-data/troubleshooting.json b/src/current/_includes/v24.3/sidebar-data/troubleshooting.json index 739e9cfa5ef..5ed2552b335 100644 --- a/src/current/_includes/v24.3/sidebar-data/troubleshooting.json +++ b/src/current/_includes/v24.3/sidebar-data/troubleshooting.json @@ -56,6 +56,12 @@ "/${VERSION}/query-replication-reports.html" ] }, + { + "title": "Troubleshoot Replication Zones", + "urls": [ + "/${VERSION}/troubleshoot-replication-zones.html" + ] + }, { "title": "Benchmarking", "items": [ diff --git a/src/current/_includes/v24.3/zone-configs/avoid-manual-zone-configs.md b/src/current/_includes/v24.3/zone-configs/avoid-manual-zone-configs.md new file mode 100644 index 00000000000..a6ed3a064aa --- /dev/null +++ b/src/current/_includes/v24.3/zone-configs/avoid-manual-zone-configs.md @@ -0,0 +1,3 @@ +Cockroach Labs {% if page.name != "configure-replication-zones.md" %} [does not recommend modifying zone configurations manually]({% link {{ page.version.version }}/configure-replication-zones.md %}#why-manual-zone-config-management-is-not-recommended). {% else %} [does not recommend modifying zone configurations manually](#why-manual-zone-config-management-is-not-recommended). {% endif %} + +Most users should use [Multi-region SQL statements]({% link {{ page.version.version }}/multiregion-overview.md %}) instead; if additional control is needed, [Zone config extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}) can be used to augment the multi-region SQL statements. diff --git a/src/current/images/v24.3/zone-config-inheritance-diagram.png b/src/current/images/v24.3/zone-config-inheritance-diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..8126714b54133d4c3d877ecbaaa8308157662405 GIT binary patch literal 85418 zcmeFa2T+tt@GpuhC_#jkqy)i5K*@;Ys1j5_KtOVkARsyCu!kcql2imFE{Xz54wA#F zD^Zf5L;=Z22FdBp0xBvVs9X2{>eYSkR9SxRH{VQ8_pihB^vv;7xF$`APl=C(g++Mf zvcz>PEZhbxEbLy~L*N&i$bf&rKaQGSx};!sN&1qNxs|Pw^-TjKX(LM`TT_GU(igF? z_yYq~v`wg0NJXOxvRO{o`7|daJUqJ2^C-vsdgH(C!|O)b7M-ji<~bMcSB-s+F=gcL zTYYAFR+RJt@$00Xke&_A*G#8wCy)EHJE%SK`8^)t)3epD{4NR91b&gL4R7N!Bwj1J=H%*VFT~vI>7Jph4 zZJ^>-Og(J>;w4`riY2i*!kD<*$JybvaCP~F#Of<9-DrFV&+E}g28m8$_4G2cWw-IJO<%0^Qhmkrj953<-YdAbvr?`j zr1}7|3`sel5JDR=t|f{ldmLCL{`y~+F)~g)5{dhCM5gYVdhv<)l*1-ZhM$`}S^U^D z*8S$}(mGb2^6mA?(cwnUiwPU_Qw=ZQOq!_B7a#F-F-dl4z`JEl-^_J|4@pKQFs+0@^3>yWn9_6VBgV`C*7Bn`y9H76BzHV&zxF@In-BlzX zacg~Ry;*f=i=~dBho!22k<<-j(I>kR`_smhx7q9~+nZx->o?x$4z*X z?4eoJ!b4g{ufDE+(Jys&NL!>$w|Av>54)|dt*#BN9a8#sv&-apvx)ThO#DI9aK%pK zMU7TA;WLJJMBco8V<;m(Oy>J9-c1fyN4{i|Knm@mLlqfThRGf&-V`tF#w=5^KYbBX zmI^z>#?g1zbw4jkD)z?O z=R~)Esp8mlv+YP$&C%Wf)0muBkzen+AYuTgZkJU6mK9^zkR5?av66(H(KIoPmbQp4+Q+*dM*l_HwpJb&Gp39a7^ z;s5CK_O+@!WqmU3=$xjrB|{fht9yyg#>$$j$Lzu8Zmj&cje{Eu&IIoaKGgsNVQ!>$ z<(8Zr7BhIp#lk*7iG>564uC(R2dI8LOC30ab#U`{2o}~|Q!MQ5HrK%4nE&pBKbUW} z{)WUszPG?_h&%ZG8M_zL^f0`w9{h!8eOcWO3yYiq^XI^o>!+u%u*9*hNL)~MJ}~wf zZ^}qjoo?!yaXm3tjNdZ@f}8E<6;3lM4E%G7STRf?NF#|`mh7_l9jKvf=A-iKESZPe zBwk1uHgevR!S{QVbv%zaRh(Ku_5_Kk^{G?%BiI&Ur%xM3O^$YkySR9CEqhItx^*wb zyiD&d8Dbxb?k*@;*qC(p=+3pkKLEkTC8iVi#`^ys{x4!e6rFV?PVfoL->!VP*m zi@3al$+lYR2!ik|@q$lt-^n{c&VTAgtf1D-i2uSTd>Z1z*RV+O zwJ*^8)aI}rBvbB_9RVX8yQ2CAYlg?WAt%>;K|hB4zIfBfNTa|$TEkTcF2||A%9thD zhep|Jh(qKO6Qzg#ljqwV-C(1G-}?BbPJSPX+F*JB?|e+`$r$=wMer+2)LwaGN2(AA ziT(QXuJns1;+$@RWr4_d_fHe-nJolC_o z--Pep&vkJ~;3#IIAvlgaQ46wNwKukyA14ioofL^=)n?djX5v&}#CC;GhYmnI$XmPZ zJ#~?XpGOBd!X1l72+OBT#`P*gGY>9Np^W1e*+LTp5NgqpziCia9ZH<9g_nFhw#6hn+-S0uE?S#) z>WbFTEm#cadJnnf4M|VpepA=8_NwYWb=9Wv@$!e#w+=yl0`4NRnvNy;ItkkJZ#%R( z(!#%t-fW-A8GYA{oLfqKytFbg6g#tKki0ez!v`WcloMrFji+<`kXOpLlA7_Elfh%727Y{VHvHibkU0B>D@9VML0W zaBrmBV3grPcoxNvmlX9QS>YO6!q7Vae_i(Menb&g?4?aWiA&=McMAz}6-w$I#O~DP z)jLMGso{g#%?=JZE7C#kps4_)vU$AL_=kjZA9VwcxjnYlI~yC2wKFEar^df6pZ z?5DL1>3#8R3E9Nm-O#T98Abx#T(6Bd`8&II8;Dg=98!CXE9TyB3jUiyY_&t)>KHKFtk9+xJG6fI4k{=iT*$3a zGv>5-oIGTL!y}T-17lX2 zg^BlfVu5YcdkFeFP1AuJfJ@DrWx1(aCz+?J)I8QNC8ul~S`mu;7b6>8vB8%X@Q8yhwam*YDhtW~3=+QG72d5fX? zHeid=TUILi{m4QM2-(2Y{I8wu7%=Gqu0Aw8SJo7Iebw{ss;&F%I zZ*L!e#s$oio)nzhpe)|F(bI-CWY66z?~ZUspwPB^`&CjuW*aAl zv3=aCu2;RqS0HaUN19)Ib@@kw-5%gKFz)_M3KW9v%F5|XP>UmS(kE>Xvy8$9%2DkI+1(3mRF)CUXx5;)VBk(NH2=2;3g zn_egLy0}GPQc7;~dtR`RsFjjWf4w8LR3L8s*gSsLTnWGro(jyUxnzANrvC=j1NCL4 zb)BDM`OV$aXNKxM>`lWVkTN&CYXL; zR%fV~ist}ijU=C;aJ2*4GM7H;DjVkkf7@<#-G9eih}Du&a&n_?E$sw>By<^mz)3{$ zCznfLh;PCc-blc+7<8O1jhMZyjE!p^i()dkT;E-M^8^(B5f@nD+gdSu*%gdL;+P;^ z;!PobALMq?05)9elNLwT>!#M#O0)0qN=NbYGV4&<+(u7B?@AOi^d5wcvWgK0mf1c# zNEl@I%L*8n@q$Jv{-~t3ELoL zD&y_#Pk6949pNV&&NDi?n$JO5mN$ZW+s94fUfP^eKQq19IuZdDC>28vV6FUO1;yPU zrEgw{n6d4U>vxz9`=i(z8_NO~_0f5ftJY{;X|3ZAtX`b$-tw+1%BZi4G~s;23@2Ntg;k;!El@bM*6&`3NLNq>DPA$A*vFzbwP{ehCmNh(&uokM_2 zuF%{J!p5jU$5W))#6?%xkTzDN@TTF~m|FeXy5TcV`?=Iw&%Sb=B(a;{#7D7RPxmaf z#8+wPtJbb;2bnU*oRy-Dr)CFOkxOK$AYpe2;^Vc(sciJ%hpO&;IE-fqd#@|Cs(>~i zYA>|tIHBldgl>^?r|YeU1qCd=M#wt4$fBEIGaR#bW!5FVN<|oJ?C9qu3?cPml1HwN za4gjO1?tjHry<;DWlLW%k~r&-+9;wY5VNpP>Y~R3jmxs0)h|f63&Te5*3i6V3Hxc{ zSm~yG`3JqEsJZ(xDcSg-QHFDu1dT6Sk%+D@fI%MReUhy{At+tjq5t!LdM0lzIgWf z>5m8!f)243^KQk~u)b;oq;@N>mV;wTzh_gIIJf(If+&L-3t@b?+_9fIM-GkJxcvq3zd$KAWFbeEHnKhf}$_7r(N-KKBxQiW6B)Wo=ZBN4ik zZI1FOWd$#UoD4f|+(^IYW5p8Pvz%1K+dGA*Aj?hux#-Av0g0Frt?3}-=v2n)gJqB% z(EXA_R#Gc6|M<8@X|JPJ8!c53Ri4eolD=yGohz~NmucZq@dcVSaxZs8Dtd2mj-KEy zZHii^@15;^;&Euj_?%2><^bNMj;3Cvm@J)HBdlt zLO@Kcl`ypjOaD~s?5Lu2*>!v8A|s5!JbvtWu`Acc^>|W>z!x0jLRv%ov2R=6<`}j3 zj~cb}oGbo3_hAC0_8Pd~!hoY(KEt`_(z2EkoSj&YUC7q6{$YHnv8W9@!Q)O7Rp0rg zmDUxhYYRaMqGh~iH{OjtJAOJZAi5XzRap1SHyRr>A=IavXfkMkgYQ9&x|b!|WJyc8 zKbIz+N?)WRwBy`a5uPK)@Ed7I!fr(Pt=t_h+ zNeXYEKDc`w>3YoFtvJ8w4>vtd0}cF&)srPP``O^aCKVB2UBnk~41=7!g1+4c3OY(W z<*s(OkQc7oCM|kd4YnA6j{tAlfaF9eQq* z%0%aQprKfoKtg76ZcE!yOB+kJNnQx&xxm+a~Y55hzz7T#im}R>d`c=sncU zSUWgoz%`kz*%q=n?C}NiNK>()wD|P)(+wx!<1r(CK|hTm#x8FKJ`G0j$>6&2dzD^| z%Te?+CAsSL8&6z^D|1y+Vn0NGD}PfJ99+naKj-r@D(Vi&VY1pG=yZYcGt17~rQ?h> zBhb6lYLV*|6j9niF{Q~OKh?+liry4?FGMi3kjtl^NV>|p&5Vua63WvICt zp{>sIr2Ex(L=C*IK$gFehMlCE{(51ca_B{-=iuefPm__BN8KkX@y@vn1wz|~YzB}~ zgz**Y&drm@of`!zePNT#6}X$1}&nG zC4JyWFGljfk5`k;iktdl zhOpXaGj}t&7Q8+^i;1*wNt2g2k~dqWCdSs=618mCm#xf>sHX1jY4y4{Ap6usS7~8v zG5UG2u;cOyLN=+&&Vxb!dK(#4_lXDNpUJpPHb(W;Pwf(d<}gelOwM$ljKG`N+>v6X z>p9$w(?E@J!}!?3w9JnUFP&EK^IlPBAhCFi{fVw;J?oIYoa2oi#IQPJmM}f6tZ?Kp zLgmqFbwY)tyD=Kij`zN9nY|;Len_}(0vlIY88tikBG5naPJ_wC!ml*o?4vb&4&-~(ksjfiU5)mZ#^FLQtS}um z6nYO`8ttb%Ev_dCnJ{Mg#{s0~atq4xaWU!_Cj|m^N@I^Gq}$hM$M`Kv6@}SI$R2WP zY#hIRv@?4ck6IctZ>qCBOBoNEZYyBFB()YxwrwYj6lVc}dMc;czZm-<$Oz(+Ogz&U zLW3m?Cbir8F%GCQtuK^k8@CcHN|%rMq>7n~T7mb_+iE`^U;t$RRcsf`e8um8zP=|U zQM_RNzybNbegI#zngF9fB~n5_G?6e^Swet(4aZCmEEQsjHkS%<*A6UX3l4ow!UN&u z`iJcdcAObV;AlrZa9lR%>ah8=y_ebW6s*C?GkNw}cyBUL1PuH+ibUk6$Nnk(-|=7q zGJkUZt&#g_3ldG(krTCkCwGb6f12Fwv2xCU1)!w4+AQ-4v>+w4b)0NdOTg0*&Imw&o{cTGVWZ*lgYd>pPUT1^*;MCoFxk6tj<{UzaAt6Z(HT$Huiq9TG z#n|JSdz0igKepg$fT|}|4P=-GCURhj&GYpdk71V(4~-xigembbjk-WfVFBR$w%fNCVIDr5{i6wsf;|%d!z9aSr^xG zm@7i>Y;78VUD#^KK)aDH2a^Q1JR9HhFqAX=!mWwc_sP?D+kqhjO3Dh9^mT}t2{tnc zP*U?ZgyG_tlq;LQ*=9CckRB*$#0!&G^n)cTmVvcrdzPa~-UI7LoFt+5b@rwk1F`Rw zy)d(BIWT~%eEM)ZGWahv#E}CHdi6fX=9dNYUp^!SNlH?6<^42mh5-`7B6zRu2Q?Ut zhxuRKPg4z$U}`?sRK6cRC;;_QGF04GQ-45$#x2)Un+X%gZh`Kbn}H#mkYdWcbqbGH z7(#}Zk5oC|=6f~H=)gjF*lV6mvl~M^wK|GEpVNgm+Xb>kL&Vk|b#Nn$6;(dXIkyB@ z@HF{m(36SV^PjmZxEwbVK(&}YR|R-4LSEtKBSBzXekINW^f_OVi6ej}~ zTSYee!Z9^}@0A*jR-VnBP&2TDqdAM6^1@}dl+n4i?leG_U)41XvaSJSU4Y0rw4h}V zmcD2+o0N6yz$EsuZ$m+!MhSr-s6m2!q{TWl)hW0GUVd4Mj7|~wkS^mIa8C|j)6*?x zOGVw--&suzQ?ruFZ^NPOoUH8-}4aTNA2b5CKd12H>R%W zcNYz|dUd8v1`+C#Oh+Tq13F$PY#dZvsD7{w!9I$+s=2z93H6W3TqlI;1}yeh=9bWpTBMiqQNFLVbwr02#O+Gm&kn`& zFuU?@p5KDR$06a^;^o#ZDk;TWurkV#h__U%Kf{#W`3RQ;8o~N>@Hhw)8Lh9|FZLr0 z=x5k?kbe1M`k!WoTc`UGX0TG?%<6|8!fdb=CkymAblcVbRWKz2~>7CfddrwCQ&sKw6g5Vd9iWyyrXqpFoi>bJ~FGPOQ5C&>js)pALH8;j>OOeHP!?pWwPbzs?MJ!K z6K`@QUtY61DhLzhI4Su0RIM%JQokyn$W4Ern&v9Wqx1KRU)deNw4{YyGXuXC1n zpDa%z2B=`R8v#Z1rJWM&#-9tI@dMrQu)G0X*WfU#d-pmWn}&TK%w-#hEJS^(U)4zJ zoJ@u;g(`;V-~UEZbQb#)3^-MC$G~HB84wU;lEn!!9Dp3gKC}?NdZO&_$&g;x6Qw5; z*yM|JRy}OB#fS+#C%7gZYTb0DMwiEGWJfUjXzgUcj!hkkbYzk+!^X{Fvc7)aL0w{f zw2lDA?zd=;_JFE)kxj22l$l-Qa^`;DBRoqZ;yKw;k$|KrYa`#mI9Iy+x+{@ocLP|K z9I`@kg)F1R3u2jmDV1U!%*u0r48s2(+{XQ;s&{)(#p`au5EUKgW+uU)){AGhj?19* zWt*qv%@t52U`(1T5--3Wj8u+2MwPwB!Z=Raaj36~ukj3aA$!kK!`z7vVi4?GWY%I) zD%GAlIww9uuvqeP>}f6)B(2`^y^jt66Ntx^kVG9tpA6Q(0u=QS`WMqpR` zQHfDEkU}3^8DA80_1>hPd;T$8a^US<-e}?)j@zHQMU{0m-lJnUBn-3axMhUSJ9tqd zY5knd(aBMTjloLs`uFp>3u^^|oPM(GPmM5F?8#s|IcW`A*{k#rJvB)8t3*EvmM&)M zEk#TQwvyO5p!1|7CBBA-Fu2^|7E=Q3yvo1xyzrn0JoZJvi0VuG;Xb-10BB2|VFPEx{| zev7U1FA#~6s}krO>`3ExqT!OO7bU`EQA@Q5ilOQw19O@6G|+;LJgI(>g*rqdc4V1< zVTE)tPcKMY9OF9U^zO}JnCSVWslCFS8sN3Ewd9TIF&pP)$K?#)Utec-6y(vDM_+-q zNEzzOh0&V!qN*!tmtM*E*lMr}zw|Tfovra9vEesNXQ|vpDSM@B>1Z||rYrM&u+sOH&-emOx{O*&d0NG*yrZid z7@9&e1Vs)PI$MCj6V>b8yeOwX>%F)9!$P~3LQAn_Aw zw?TIVN_ui2@*CITm+qjw=TT-Cv{o^`tetsN8vUft@wgHaPk4d1<7eC#hYuD*o6+-* zF8&}DIIL}y}|UBh|1HC!=+_emQ)m5q*rCo zcVX-Fy&u4yVwm>H6HLW0SXm_VimL-U9M(l{qx|p1I10>muD|JrD?1Rio3N5D_e38b z-6QCGPvCq3ZATvz>@0=Osy<=!l!d;O7wr0ak=q!eiKGmVH5kBR$+_w^?ev}QfN}*; zN6OcN+2`18#2@^8e0nOOb+H)HA?3JSqE^P<`y86?pD*xKc>+<*ByD(1F3d@yzlC2! zq&EeT3=32AWVrUQiaR}E@%a;ybz9aANpu@jAgKGE0G2sODr`W7R_BXX6%bT@xV{uZ56jmuN$l@ znn1BmWGYb%*Io;R%Yvd83OCoE|1q8le*Z?jmm@kBp$(OmG^|R$T>VhD;(VHWe#V!t z36@SbJ76k_e1~bEND88up8K~1LfmJPqhdL8YGLsG7Gf$hPtkBqY>-SR-CA9e3KxnR zxB#nt8#(u(qV(cG&uHvP_14_WDva}nRrMRTdLunzS@mt*vdaadJ+-%S?WIww{0_$K z-!>DJ|0SqsQtWU^!>2Vk&~D-fcxNhjZA{(SuOG7VEwFmv!$dv1-0JxL(S2d=0}Tn5 zpyEJ|Z}0-gfAxnkMB@0u-1^cIrSQQwh{l=W#(Qs;{S^6}``+u#(nsE@Kf+w@e)O1m z!T^c;O94;oF0Mo+xj0%3q=Q{P$n1fuq)R=IZawBB_jPk`j!{gaO%*iDfqkks|GJ); zVUhneyS92{TEN+?c543h%&Xdqxemi|ADG~!nCjQj_al4RO7o{6aJnGkWN01nY^EG> zH?TSUhAw$15pBEmMb@$;b*~i#4^u7@r&8hvRf0>r(KZ9}25+}EVYq~1DCd08FIWmWhV8!ZlJRc+PX?2hJ{{-%H|rH44Nk~7UXG!>h3^1E+6gy%(a7ZIy60Ur`1d;hr#k4M=UaKC%Ypc?u@O*t2>I?Qdt~ zJG0z66oly9(4;??*Ylr&Q(O%&7=^+@%bPCyKXfVzHs#NCKR~@&_uaKbsWCtPv%T#I z*_5godO2#f6HD}Ej;0FN{O?Hf*vPqix@6oeulF}73osq9%Y3^aNe!KTjEI3E0~f;) zvTqb|zg-0Gfg|Ax(_VRub8t66)B3%rg^jjf!kL4wjGsU0=y=?tier@^H2kN2n@9^xAK4o6CjnTAhaxf?dekQ`#}m z*2veAXto(7T*WM(j+tCF|Lml3xQ z?kGFhC4||+z!|UsboNEk9u=6Iqb5lK5_ps>;xOF#YvO486$1!e7|0y<)AS?A!MwQj zl5o$E1auvTskD3cbU(*e;wS+LA7x*`_mixCgng7nm8GZhFbYQjC?b`|H0Z1s` z5zj6Mun*8nxT%u78mL*1_{5eTT;kz4?-k3#Op@UzNv#3aw+`C)Vkd~wf{f1PnqD6W zGWdDTNi8AgFUw~;h(vqglog>yzidRPXmhyL)V|mc3bw_$OvgE=OZn|4@$z<&r?3QS zcI_1P4UD4N0<{h$G1DUes~)x<0gSY$SV4+A!^7vWmC1etjhz4-T*Otw!+UtoO(x2L zQ(M_p*=FYZ=~)RRP-Pji4(>xcP>f)O*w)|1!@D0UUI0}}IM%kGqHMhoK!o|s80B7N zs2B}FDFbjS`Hk(RMwrQGQ~@G*T^s`gK}8z?D3gl~`a}(Xgx$8P#SphPe$zK*a*Ze1 z$t!RsqB|KJUErB(ol|czr(yxjxwW-S*Ly;s_0pU6J{1pS zuXt_~ve5yOuxd|%uad-pmw8QP!kmwPWcSj#A6;)g=oN!4pK0FQZ9e|`@;4P%rUQoO zpDVl)`$5e#5LkIf>t2}=%zz4TKm$Hw8}S}7z$Q861k}8qA#A%6IvkCPXpD~^8vCe{ zxuI=$(!l8#N|P=F<1HCtk+jRlyy86qT)wLBxq{04Jjz^x*2cFF&uue?3?I~2;4^)` zzYC4u)OrXT7la0fH*oC6%r>hBwq*t6p$x`+PGZ%%%b2dr863-NxhGcETQa*38+W!? z{?bSx6R_nTQ+LB|Ln`Zi%4^VSZJHk8{YBw(SgqA+#!Yv34w_`gi|-7QkzDH^=iVwc z2L_A`sFQ(kDreoAqLbDxr>VxX(!)Ow+-k&wlIHP*exNub#oVu96IIp5q3MsR7TI*Bg|8@vd|w^ZV{zCnV&^P z!5*?+bmH?M*oY3$Aqn{ecD)1EHcG)F<5Rqb&@UX>wiK*NdLJ}RM9B%)8qZPLv2V#Z zjt_W)SRLLcc`fp7q7w@L;R)cT5FgpO%m_r&H9es%?llM;s9&g)^EFa{9Y_H=moD6TX}=@jf(PLPa~e+) zuL;IG4&~?iq)Tm6ez`;iQ(bq=1pmxSKs4v+jAZpLZe`;f#n=^zk8k5ozwgOk&Hwnn)EDpD%6cRv#S6PlYN z(SS8d(*wVTJu>v0@Pp3*D%69E_S%=*gIvEAS)$$*A8n zWQ)NqIIp`PZ~tH!XmcPGJ$n)lFXcpO!+Q%&jNje~Um+*^QapauSQ{e5WcF>pNr zYRx<=I2EuwNd*+KeTuMh>P*XaEO`Y&Xt*R!XF5fwomMEA&C6&^EUziO+Q(>M(}^P#HGX??J&X6nzEDY^8-& zDv>?eB_8(^N_*1lfZ>A7Xx%t>r4*#!$t=>IU(5;oa zvny@e=PJKRgZoy*?4BoLGV3xO*R1Fq97eseC45>7`*okqxRrm7_(j>Zm^_uje)YGHTMCp_) zVbN>r_YjjBM)b8yNHXNzw0xSxUmPhozyZt@XO-`tN52(PQEXf@f1YBMTnghKiGq3R zkuf>_RsfuV*Q@z9Q$uS*#VDrx{-_`%hTK)$%>>2a z!;1_aS^2k8Om`t>={?zca}n1C&N9mf@$>3ip^ae^!Dm8O-O68g^BnCm57AAvZ?3F! z3Q8eLuWRa>vhf*^u6R=gZIN(4rLUo$Q$pQx78)Fo&(zxu6D8xSW8@W4`!JJ(=v-(= zbkLLT!rvGQeiqlqUN1tuULM;rN}dq#2GDDjnG7r!nxKD?Ve7I(sz+P zu!=2UfRTv*I8o|bbJ&1kp~hB9f-&UBiPG8`*f=3_a7JzEI$B<;zuW0&!DbF?yH3#P z0_@y@kdqqfqTfbU%H0Be7`D#ZTZs^63~`xROq4X#58Ne>78e3J zXD1}UCb)G*Bo`dIEoBfr3@%bQ1(f@!oYdi>nf75NM#D#4;vt_sE@iKI6r8(0)9%`O zSvyTs6jr|k&YjYihfWCFnwt9fNfipu{Z&pVYn#Vk{cmM6uU(lYH6s_LHl6l==qmPMX3s29?jZRHqPii{0@rL_S}fS>H8bi+vm!*Z6c`34DDZgQL)yyOzk|vW$UlkPest39E^r2eldHiLTTCch-k%u?hIw)nj*oz3jI1 zN^XOtw}Rd>XfYlU1!D|tiVBoQ552o2baa*b#My4rnSY;HkmIFj^bDKyP*Iqm?Fs{K zB-iAxn5kc319bijdA{kOlE+=e?i|_*wvHYpwcILemegxU_0`K&~w4@t^=ZtZHY*zYi3`cIRS#Ak43b#!V%ZQYinAbM^{Du89y^G?&#aYArL|C_`0Zqk?< z&#+@!^4hgmsk5GMXwd*K2oEKYT`ABTplQH@i{kZ{89pH@|IIh`tEYXNfoW_iEFcB?Falnh*9?4U-qF z^4h!XdL`uVEv7?_`D((sR9hk=$jBqE=R6hiQ?PxpV+Cv`kXtA!Vd7pr@O-b-Y1bEh)T+>tvWttDW2`gd%-vL(Oub(bR>6;z)BFOx zsHV{fP5O0VScpv4sxsQ*aM;O3gA?O4Claeqhj#l_N+2mD`klLJryCJpvO?!Sj8%d8 zjlxQ)A6(%Xk}!0cJxF_aEdoyzbB^}*K&h%k=dnzo-YZa06weJ6IJ8R{yf_oi49A;^ z;L5VoN;@agz;K`}-QJY9uq9mfQ)Ra?_qhX%We?AG``X$_FjmMiM>J3SAzn`}f!cpi z>{0SUj8Bv)R94T$7%e8*Nyf$B((-P`1Q>9yV3GkBT9WC7THCa4s%a{yw8W$8r{ime z(X2Fw4NkaE_!rJzB_#Yf9xxzVT)?Dnt>xJTN~}cNnC_smK=nWv!^jr;<2E3k?86(> zlm|BL(Qtq_Evq0mncxmbghF_Ct4fKdu&dB@_s`@bR$ZU2UVWw#Zj`f-G~BNQ3yOJ? zt!X;>z-HauOt$nhQ^1{6K> zJnvSb@GJJJmcr*pR8yGSg0?QUX{O6aKG7dbdB|+%ZCT&00uxjc$sPj|!{XquC{e1S z8DeuuZLfo>$B9A0XvQ9b6$E>&Zz)(XNW$0T66YN^DD#fXSE=E4ukFWwjXn?Cp zO#(G8ZT22Fi-F22>4;u9mw>=g&y}H$ejo0JQ^~B$0$3yKPqdo-Hxa23d{GgUY|Z`7;n7{wa|^NB#Ym z9L%PMgDv51A?WXI{~x`;X*b`qNl+@tZJ5Sr`RpY)45PHcmQvI6wg!ql#S8+6PawOI zb1m({e%i8Qt`GcRK1{GTfV{;=0fID`TIlxE_Ww&T-Uu#ngMS=jCBW%CY5_mS&i(8| zBi!{kwaGKRK1FO+i%PBo^hpwyToQqb{4i?$FXB-?*Xd8$I^jZ?PLtL>5dT0|&355_ zlEoeD>mzrLo{l?*PjlA18h&wPyjZtreZ0MB(yML5A$3lK#bro$qsX#*!QE`~)qK}v z3i7WSya_SwV+sH^jSb)kQQ;ZuIMe3W?7Y{acG&GC4OBgg3Q-xR#!WB9d-SauCBc>x zaezz^yriL1ty>fU}3@{)XN_UgyC>9gRc<19VPoMGjV z&DRPs-Yfrn_0_sd0aXysM#Nfxa+3=^&P`PYJoYzecPxX3L%orzt~^?lV50;63NK$Y zC9oa7HIpHnP6~TZ#Q6*v2tF`?lv_ure-8eGik%M`jt6kgn=v_w@1Sz$`<*5QzGHrS zbPF6N=)a09mM}>`nG>5JRRqqAQS&&_fn1DJ~N{LEON9xyhAvDwXwjoLL%7% zefL$flA;YD&|XEePcQR7#^mS^&&x3w+TyYJ5caiTa+=tNm#J#GUT{LRC`66hg8 zzxohCxE5V~+E@32Q%4s9SMO(;k@9l~Ib5?XRta>oO)9z`^lQr?Azj9`BjJ-*!hTmL zPRX8PHBV!-q|&GC$Kd59vg01Ktlb3J{DO?g)Q-BgAmbJH$Gns7Hl?yc((N@d9HSz- zolI8qM^O%fV%;6xBv~%qOtyUNT3{>AMqo%rtd62(q*f=92D7~fPLf^F`(_uv@(+7} zFE5BhNK6F9Bq+Z--Qj;xuNvSw&V_Eg#+`XJb zLw6r)$D;ymQm(XoF$!MENfXdv2lpJY-#3oo(6WUK3yr&1dHU8Yoc?1E@G&gzoD=^2 zci6Z`bWjVgUlU~>1@2Ri{Xomzf>!e#p$~@EPs}|-3$JX2)Vguz=SmK??_=u?^aTZK z5lNu=3R2g>+Ey(T zbtAivBw5gV4a^{}4^m3x=cDk zt6)sO1vUz?53%&D^yuBafw`?b<38{_7u|V(UZJ(gYBT86?25)?8pAUH7{2EoF7Ky~ zf-@Y*kusXCN)65m~s z^1Q76y$^6xL7ji#WI(>jgP2x~X%Qix^2n(0NgM$bsXYN;!nYaUF4%c|hO?09Y*}_q z@9<;yTuUKiq1f@+o9)H3@0QMuQ=4))m8LAXPcKBrgf})igw$N4n8bYsi4_mYy8J7g zcSxFsJfFASk=3-@;Xx5=mAp(FJ4Nps28zn;x=rh>J-=itMLoE>s*v@AK{dn*5k*HN zSAgc!1I?Lt`Idum0Ldgk{bEu@wl~tj?)hEW`Akeb)hqmb(SG`R2B!xM_wmPMvLF8F z>c1YCL6oFaB|gfygWA8f7iW+2W(3{;287+fk_2p`9K*rXsK50E2>$V^06@P}a$g4j z7*M~^BRQ6T<@AqEv2pu>4X~y70f8L@?r5~B3-GJJ?kS&ZDBSBDGv>ql1ZU!^v<)$v zf5-S6-cwUE6l2WWxdYvYS*WZ(W9(%BB#nssqI%%gB-IMNSMEO#X+H%hICQFK`92YS zIGCbXg4 zK<)L8gCUmqZ83yA1M|F2jydaSPGHVs$CS%`3w)pu?mg+_5SWMAVbQw*{ImM=luuKkQa zIgh|sB={zT7(RV$;(G@dCAV?bK5S3@6D>RYQVT?#O>Rpq*$D?d28ilA9aQ}=q3%9>|MVE5|W3k%;)5W z{uqCm&N_}LuvrRXMxO6PJt>-M&?PpAq*fF?dOHm1f6af+leagKeu6AB`~~$W9&~^n zIl*`Ht!==Ea+?*g0(G5b{oTC|oB{v+jDIh~$-1;3bbg|QgA%E~&7}TETo-$t zfpU2b;RE$Lkx4dPg&3t@=X&Z+-JCxnnHlG9-fRr6vz)F)BudCeog`;@WY9?pt}+aT z8i`p2n^4T6JBkv9&}Gj9kwN@(Y47X2svV7%G@TMkcB_a5K3)>6ZdjR3+_sr2z5s%B zM(yL%p$I++;nQ!xb?=`c;pH`6e)*?+MZoF3M@%)~hMK7AmJf{}i>yPTc+a$2TFQg2 z_Y1Ty0eKGx8SHF}mWHiMqc;K;wa~NlqU4IPXPRQKx`7Kbbm?`Pbckom$ft|Iy#tSU zAOB#&E)OYkJ;nSVVxSbMP@0l~^lDcY=ONt7GMF_!S+fUcuYngFa`;5qril%@>k%MsQ8!o@-Cg74( zQ685|4AO@HmL7xKl9nzp}&c3R}Yt;_K*IcD)6C04G+y_%*l zC>1`NK$`m~SnIZYJ%W!Y=nQR{@;#mQrU@o=PWalzVi$e13bY7mZl~GbMXi|7T|C+y zv(ZUyg(OoHefffPMEHDf36$vGVhI3-?CyBZFoe&@=xT3j0jsRy>K=r_Jqf&Qt$Y%BXE&|0UCRj~N1?RGO z^;iGJy?d#<>oUKS9Pi9untA4LN;`gjS#&sl3{ul90Cy36Ga~-Mzbr^=NpR_%c1_pE zevZyi+7dR}SIW5^YtU;OMO7P|IyEhA&+?sEY56?p$RDlI2A{#PW&{a2Ymecrlm=j0 zk_-@{c3)~9Mu^~=tyAH>4~r^~UB(fHZ#xAMAi&D%jq}|i+7x~@j!eeRRz=i)o9k3T zUE6SufsylW-Ib;M`rkA}HOhgSc;RQ5wcej z|02s*A(>prHuc)1%S%G|l>RrHtIyvtNk2;L4u}#xNsxv(bLTb(RrZH?dL-GPFx$)k z9`6SA^eW=qnu~;~jH&e5G_KtvaB!>p>f}2HpZ4#ECc!P~&#ebHq*kx|n^rJP32?_?xJC9hUgBM=h>%g*io9jYj^3_ zdG*i7P;yTOx6)4EL)1&;BwD?T(hWW%De$~P6jy3g6qXpr8}Wu)9y-Bi@}$4iD>faO2h}>bV+xH z-a*8?@;!Ucw3y3-8>ehNeP#5XeeWAU@`ul+%SiIpEoJ}3Bk6o7VKjxpy9gdZ!s~0> z{AtS_Th4P0v77HWf3_9uZFTd%*n8`+s}AD6i^XVN<~R&DQS={ zx5ZXsi-2?}CDI{X3bK(#q*FqW?v6Vbp=?CuzQ6aq&v(Cj{@{7m!(MC7G3Oj(%+VwE zcvD7MaODMQPCjf!++v0wGs20n1IJSg{ACLj=bp^5odN)?#93{M02#-2~ja@^a z0yUsk4yqN-Xv}NHxvUQt7K<@@A)BA01#0I|2`LGyw;-`N21woB7a>Pq zD~3xz7F8<|CwL$5rMY+6CCVm{d>ae&qPXh~n>6!+xl8DF^Q&1PMZz`qgK>IFHEm7$Chh~RJ|cu!?p@`xJ|c;hp5*OK}+MyVl3KeUO><@lx)DH}9h z$cA}~V$!U4gNGikiD253`tx=_U095E#0*sJ`WJhz-i$H#gsj8*3e%Eb&+J^3!gw;G zs7LPErU|74H;BW@llm7QZz#i(M>82-oW1jYGDXk1cz1~u-U6`w^O^3&-&m)TnBcKm z;+^_dv*8nVbgzz=JH&kS@oiO_(%Fz2K_(4Aq=7=c&9s%>@JFN{E2RC-mk;b~gNSmf zkcP6rO>NHZB(NJ}KZtnpo0U8`mT5ixwHHCbm6z?DyDC}SNLf~AFpBWw3dYBEX6T+p zINwq2(T)NVX%IqystoULK7prHAK?FGs{9WW;V!Odw%>Jz*tMtWUA}w8Dk( zCYli@VuI9~Ez?N~FG+i%5R8-wZgfkw_M-I1o6er4J+>~DTF$+Q~^F`0g0KBC_NV&jPSk)eGn)NCNC<JP%?8DS+HwFyC?BfFEQJj$Q!N*Fl5H;_=``C#|#I;|I=p ziW!3Kk;}DCTt4Stv{IF!XwI5?8>)^GB=U%PYb`r?k>us(eDu}s_jn%PSRUBtTvO)L zJQ+G7ULKNRplEAvCrbaEo`QbnGlC=d#I+E=DIlr>X)ya_Qt8Y~heP?i-JMirjaLQ8 z@y3<}nY%yrI~fSTvoJxSQJ>#imlSSvStdWANVA&!;J4-=N^eL{k@Yzw1ZF}Glpm*< z<9TV(lMnsO>R5~K(~8{!QybQW7`yVA7#}NWR;{V%Bu%V-N=H!-I6XNrh{c>d9oXrb zDfC`6EDCJmv#<&piy*rCtfGTwGiW}76buQQm+ls11%0_BhHl;RL-ie`Qm(l zxrZ>PW=IUFNr=9GYcRo~`6XLo4H69Q1H-Y&b&I<-BRj0g#sH=}rb zZYCNxusUBVkmoG$^=&LJLn3X|ip!~3+~`mYV=KBVB5X5Px>{>nl>Kr5Hu@4JAkO;t$$771I-?YA_%K`?@Rk*LqeTun56l z@D*FO0P1YnbU!STIg%_gvqUruiIIb=UuF+9Egv97#4KahIR5G)1NUkc)+$4KZq$>> zS}|>YVNGww#5vZF)RLV@$K34ZYVq!w8Cahmg}aTID18DEPaYe-AtY=TxXu?L2t;#P zx+f8zK>f*m{v?XZlCf;`;~y&}kV^Ga@q_Mq(R(5n0mU&88>;-qpsDbtu-RgCr0d~Gh>(V& z;*F>39Ymm%PAgS{M}A=%4S%UH(%+2MA1zE_QbnU<(g72>YNc@K~^+ZYBbwcnI(!$$JSQt!%|p zX#rB*l>w6BsQVLqFi;pM#c@Q15X)$TQY{Z;9*MDNn}H1IbwCg^KlPJ9ccHS)^FJ5p zn(&+Me3drTtpJj97z?nt4iTH=K^E6I08i=_#r1;J0QPiXan*Khfh6k3ki`}Gv&D4+ zlt=&1Ew0sZ@G)H{&bzWmYd&n6r0IxO1BS&CLyZ zu0#FZzPf)HIQTn4QaLRfbK>W%ou7xkNKrv^g69lR^_e;m{?dP-j=<=@7(EJHN{HY( zh(Ib%vMmZJ0I>ST zV1$Rtw!1L-)K`I11YT!gL*Kt?Lzh9r1!6c!&rAvqaif%?kg;rQAIGiCUJToYg>^4d zEWesZ-#?LL!&|igT)ckC(;kChA?r)wG21%^Zf>+A!T3^kJ~C_;3hm(r6qUue;iyPr zt-P|AqXWGH{vUAr{E;~SYd*EYOI;$I7N*|g=p|4#kiM`4b2;BwYx*!0<}Zy4l<%UV zoyxcAYkz^1VD?=a@!czof^nkQ+W9ajqpFaPgyZ^){)MY^sY6iV63Sbq;0|uJ* z#20i;y9uASqk^n;UG@Jje7;5dzohs*DC*xu(;t4U@BjMM!)HpM^}g_I0DX@B#6|!5 zVW0RT&%NpjY%5A)O2R+b%`aE>pLxrEjfk*>qb|s5CpqHSYMs}y!-EehRlDUbtuZqR3y6oN~7r^Pve$|sNAxc26dNs{}V`<0@i z91npQugWeL1#DkB)7<99Wa`nngltyzwj5zGrL73SpH2u=$dm!!XGcFd2u}tIPst~R zU+fGT_*WkQEFczB+#WHz%h(%3l>ot&NJ48PXXTe~tsf=mBMChIgxcE!f`5r+gpA-| zncT6^Ao~UM+>j|(!+T~&5vT13@qHc;w~P^x%!R+}kKfZp&EB?nZIMbyNPGd1nE^+!XlX{zrVD@hc2@@vVE3i0ZZ8&3y1&C1Y-W}fu zCwMlTPMu$3Y3WemH6Q8la^Y{Ai6W0RJpvCG=dSMB#)8mCE0H%-5&Zr zPfrolmBHew4~6Q={GQbS_Gt+j#q5`7_Ero8BVbTwkHtWMQh#>UZVv&?pq9`+shQ|K z))G3PH%RWB_ZtiF){1Yi0?SkqIIA}xAwYJ8{;%&J z|M~$G1HlN$MyYKt!L|k6&V#AP4Z5DUz6BLDraEy}(kX*-y|c6n`t+@R=eH}e8{v>p zusWtOqL(KHYg3j2%QQH_d)C9s<~^*vx*v2q{-OM;2&gSDhakN+!m@zD&NewifQm(F zP<1432qWB#RV(yEE&GCDyH%ByU~7L-nr*-9ly(c>C)H69gP&IIqCZtd&|jWr5s{ZW zJN2P@pGtv6qtFJo2ZoD+#Byg3{mvZSskfcgx-`@%YT?jKQ=&O;uf80sd=RhPPxfvg z`=fTWt2Geix0H6%BX9#Mi(&F+FX?O+b6@J;EL9&j2xd}cLR<0ueD0nv92=TFbI_^k zW8P+5Yh){nlT$THd?QFQPE_R4*DDGF1+qOIAX{cD;{-T|+m0lxJBK$;!XDq$IbD~7 z1e^zGF9B=|ejBz0xTC(@E*YLbBa;XW8O)ClU^V+=2v5Fo9_ms&OL`WNbTjWz6A&qL_L0==hqLQc`Qc2m4B3Y zcGCxTf@Kb<4f1A7c8Pci`Z0Tx#N9Ka*o>Wrc1E3M-`oREcEznQm83KK=5!6htxf`% zLCz`ghO=xpo&tUA^KE2=1hoqTVDJ1Xi(9Qp{iF(4x(uZow%k{75``pLH2l{hiwUCJw*riJJ*aJj^S^14{daT!Z{6HyKzmNXu8Vwr z;RM_01v15zEjCmE`r?|Np*7g@HZ@Rptr~9Vw{RzrWZB#EVY}Tj0)!q7d;WK@B{{FN zo-973ZveL)EFFZBoz}T6;I0o+`4n9@@3tfsz-@<~TS4*U(-IK3W!=GTJ8mN*|6^{u z^HSJiw%d5#9!!rQ;J=O5%BGV(d05M%wKGuj9g6ZWQu*AbG)u^8S~h?2#cyen?OfLz zh1u6Q<>jwdV4Wc;RUG)jLRO$^1U8eF?(BwB_6{Bp`dSe~tiQBifRGZ*1P?OD=;mf* z$FTHr^rCXAnI2r7SmcDMxzfobW>IN=x~s&r90MG(t1-6jzW!0> zaTb)LZges=3#y`Le6)sLGr3=p>dxJ0QCqw=Yf(P>r~?lK&N;q1(r3{1b>H+w0uYRP zF3jQG_7vxzPfsDVu%P&5K7|=uH!D>-r4W&sW3jN9H}U~7PaYk7x zp|`N-TY5bDv#@oKn#1QvN!eQrAFlJ^-W{_fB1xIK=|2~5rgDUwe`Lb1vkFD&pc znHN(nw!w^OMbu3|B2g+2tHF0!aOWn1{Z)8T7?~XIB6{oCwbiR@!-!CK8!b_K_2#oG zgijj99eBF+ish+5R46v(72~!WA#zhO2<(dvyxk@sT1zX02n4W-iwW|*5|q;6%A?g` zLCnV^8bx!tG`UukDwd6l-@+nm3PsiTLlLXil&UcH7=+P&$NG)d2}#d@cx{~^w(h{^7N)=!@0|3UoLQx_6}8M_kCmO zL(YcB98YN@zb16%8$A5>lwE;M-g=OHVB0|7Mv%O>VV-IS)7k^Mgo+nyhQdm-xhmyl zt#wpaJS7({#YBsFe~Uv*76*-q1(9h`)?mReCV*2N{{UMfzn1^FWx~&SFv zic>+IWReiSGeH%dEDK*@B)XD;7QBHK>b=38UsT#zbm^W?nvTE_ndguWN zxN+d)AJ>J2@YgkZem2ICI7UaHwrfT5?*ueNkV(VWdq`?jJvZG~_|-=>e?to<^G-K9 ztTQ>$s54LMx@VZLw0V6Y+S}toZM_nyx_2 z_VU@JRlWKl{&W@FD?T8qoF2$DQGY4^HU#Kz3^k*c@n`Htd=`?c60+$esm)f|%V#H6 zV13`tB)Z#}f$Bz8?!4px`3T_Fq8G}umY>t;&_~es&vR0J^)H+F>feypE@+f?2!8Is zFv;!2yX&pi(XsBk(p8v*DfWm>Ml##dsy78`!Vs(49_Or@+Y#&ESzd0S*$4V=vUc~u z2tatA&)jON(@8%&>hTf1kyuJ6!S3gX$%jM`K8=D5&K}X9IoOZ?)HuSAh~ceKb>qOo zk+KZ&LVVrb9Gp6fHob`mn;XUDC2>;&rMi#!WL%des8;#97hu|pQ;1J5K6zZfrA1jI z4sUYoytu4Uj7zCXU|jn-{NehMiH%) zAtV#Qh}!8^c3Gho&^+z>Z2*uw-0D?^zj`s+b`kThy=f)05mm$UoTTk`sMXPDL7 zc+O&YY#D2s;_fZIt6UV=>fWlJac?QLowCD3scTVBWTEw4gw54YrHMSPq?cZ-Sf&mW zcxhA5bb_LCgk)37>}#MRV|#Cvd!r#g;m0Fl-uL}#)B9$yL5w{eMI9FugjBm)hB~7y zyI{w(#DEfJR6gfXNR0Ku($PjeZ2<{Zr?{|aBMBuoT)}i;(fC7-qyz!4ebw0)nlH@i z>QKyjlP0c>)?(hm!=2=ycP!|i2V!r1UIn8CAc@kK>dR!51cn;WK3{M~1Ql@8uwBo? z&!ks!UXOk{mb_#@bsa>fJ!a^7b!M9i_Z<^|JV5Cf>k;o+AmY7`D)0)(mny{)d&ISH z9XR>^p-(b-miLK%Qw99-*=~G<6Zi=HVP$u|^S9qT6dec7p9-`3*!umXdR7A8EH2Ai z+_rCzxM$WGk*~FNH=bbyqUMxV*p_<$4x)Zi(;wSH;1B{))9o}!%w&ViR9W8)6vwr1jKgYBk@oSyF7?x%H}xMaxM%7*oB>- zO+daCJpLOTcXdJF7rG`I5@K=%TSfDea8wql2W^Qy@Ab^bf> zzgzJgxBlIV|ASirILzObupX5nLqOLjVJgB`CioDS@7m#DvWaw}Xm4F8to&eH%?rNV-dmw@Ym;1EMIG5uHhH@oLChorhoE`wq&HoUbjT$(+C`H@E z4|OxJ1Fj@3d+EvceVzXf{6D!B(%;NlxqrFEpdDPrz%1Y8#tg~~;oYla@Cv(arrjqU z0I4t0)?&BPo(8w->SDRs>$->)ZOPOIFjL`7d>!nqK$i+8wrpX>g4oJmB>QgE^03bJ zXN{3NEdoSAkoki@@k@8%z?(w%+N5$CuF-7za+J}XoLi4K6LKMkl-14t^5kXHO%F&R z2bS1c>bbr_j+;3JtUVno?4QnWN=k(ckfY{$WBx7&ewK6G(p9Z+>-?M)F4zIICR&!#?C2s5jE7QfFyP1 z{pg-rd8&XY=Q>~{+aY}3&v(}@wI8gX>ByAl>E++jC!0@~S_G8#P&5YBlVcGKB1oHU#UpG(yo-X$tX z-3!{ln6V~DJl+njd4*2mggjgjgY;x%5TS3&jy(IO|0R!FKwbPVG?+E-;sob zt$q9*+>*er$k|>>yqCrO4YJ2MyWZ7v?*$RRL8b)~#XWV}Dg^KTBbb@}i#%d9*w|FM zyuZaNtdD!Xs{&zh>`8p-h|FZ&_4M*b+v^`~6mc7J`0@##w=LI9xJk}mXl#g%XQSj0 z2o^U+6gg4}N9{&nTLD&HLe?m?`(R7$dU-?#GV zygb@jsM|t@PC*HKN)dHG3?8uzmcW{~Ls1}(G-kxyKOr*+Rw*D18eD5#md*K86`?P1 zy(Q~`*3uzgqo-t?x2#MJ3%;aCC2Kt{KkPBWb1R+Rs$#mm&uzec=|0&7{<^??P}EL( zP>rP+_|3z|bZr@T3+1e!W3xNT>)`>Z*Gt5rTKLj0l)}5c1hyBB#b060G1y6 z{4LaZ>*s0F{{vTrf)78i>3=T~@*8el_y(}_xGN1#ds&|yhyjfvD2hODFSrZuhp03b z*r!q%6qi6G`ETam_-Wt~AZOKsUSd|FT_SZJ56JY9Iv~rOf1lb|Z0cLx^+nh4?3=@K zBw-%`l5Y?=sh-=t=rD_bWt$O$)=;3J-JoYPPco=ooKqo5S2*(+T4+J8V#Z-%ZR$H-i(4OQ8Wx+_0S zz9=#obe}mozG8^saS{DYME1E`K%KKtfj5zr`A@iUjt<1!OPmWi9Hg^uiGZKjpUJAQ z?~Tf_lS#0Boz*|xku?P1upNo$NA?nnp#X&}_$0MJb@Tp&8ld10koC1A()JRK0jev3 z(6RS-qW?H7I73J1{`U3!v|GcKo!`Tum=Hyr*xkD2|-s(D5tpZ2SvOe-rN%bsX##Z1as^uKSd0bf#-VvM)uDX4=MVW!oHPB`|FZm zrptLXH9Y?42h?Tv1M_w!-tztH%LoHWKP&^Et^9$QAdBfB9C*LyC^|QJCelNElC7X{ z_RTks1)19CpcyC_U~Hhof1^9Ho(L$Ee<*Aa{2&DbJW;mk5IPNREwB2_{^59<UGr|Gv%~AX3oz7sPPN6UGB-8;?-3i`)vo3fl^cD3! z3^HhdT6a#P^%raIfz}?xBK6>hK#09ZC6S=B_y2oOZ%^5uP)k{2jru*^1oBWP0qy9D zz1#$0Fi^zbJ!ueqGj}I|Fh;18K#J_wy+}1R2+&DDOu-TzS9T{Ypwxm+0?qVOd$|<# zL_sG3syO~V$sSJu5DCNHwtKmbxFGt6$?fUt0;(s5l)P1$;9eNZ{~Ju`e+-RN4`C?B zUN$r+x?KVP3oeG`kjGONYq7x<==TW|+`W?{`2z9<=Z8wYGu!Cy@tuKR3Nt%&)K5*_ zzWy$1Ia&`ie-;M=Sl{iD+{pFeI@#5K|DUUtmqB(VU`*-s!~dWm)^;8{3wUhZf5c-) z-+Zh3k9h3=*cI|Wc6$FZp@Dyy(AIsjB`W-j#^`tH(D5dvL+M7r4~REI9Q7iBxbv*| zg4$Ls0HrKunv3@ej{zpf4F8JWJQC!3Vg`H}e(M9>a)rwnt+7!8UxhCMO&HquuxLmE z{8plb58rnHh+c(K8Mo17FF!RAjLKy>rDK^w`&ryG4`A)X$RlV{jyIJVlfMJsxUmbF z0HduqW*!6yo%2o&KS{Gq$KlL@9BPr`Lr zXntR|YE$fAWB`&tqJV+Q?&1o%(Zd4g5-5s(nF?)Bch?`_`3DTY4ZV+ZEj)D~h`gU7 zS)F1GIoqi`PFnd#IRY*PIxVMuBvGGlIIw)aVt5%#XD+dJb{}q9Z? zE3LAOO7VqpIW@O3TfR*H*%%uKI&D{Ge{gT)hU;6uTrg$&I+l}uatiJKN7t1$K|OQe zQdslAmlN>kg%M!38+YP7b~jpV+oORO09lPke4C9uM82_bjS8jm*=yYR6VfD^gM~5D z)@#UMm5iP58Rb0Nu2=pC+JJunJ4nbuazSK)W9t(C&xlB(%o6BI*X* zC?hhzSo%$Uq4L0dR+yq1C{&On1gv=tx<;qbPHZ3yz;)O-fF3gbiSz)@3$x-9dcJSt z+~MiC;PVlI+fJaY3}}1NoO`_mja3z%1Rc8$nzu5q_bku>>a)j~e{Ge<#vGen1j&_8 z!Cyi;vv+Vhx(QC<3c%m%hSCr|z(a6{6B;s7`-jp5H$GQS2dq|yME~495gwrU50wb{ z1LgV0f$ukj+N??4=h?-9fc73lvLXUNr%cnLsJQ<}LdY! z?FS{~2QJD#0$oT0h_U&!&5kB1yMLK>0T713>^r7yd;@Lj?Vp{wtX}SBhIH$pmDEq= z@gU>wn~%6q#rVA&Y^FC>*;^?UR<$Qz@|DQ1qqoNWX=#h6OvRXbzS2*m(4&HQ+7o(< zQVmNUNqig9z8*ZJo!3~7wPKSapm$P*F5 zv{P5l!$=(!k*)sv#wI1ziG9+>Qn0|VcP3XRSpG%#`$(p%36^=L^tj#z@{W0=ctwJ7 zg5FBM-&K(Cu_pPg(MjnLm{g9sW^T?UKhfl7Biw0&B^?E(Cbowv>0+QO0x(w*E*biu zol;Unp@u|AKiBVP*<4=UQDC}rHA>sRhqvL{fSQ>Y`#0(Dx1JH;#{s6h4`-Axw;$;* zj=eAi;<7##o0WIb#SgT#0YYU3v^T9u!n`L!RRmaHUdFdsaP60<2EhefTF3P4lHKut zQv>ijWJz&_ahL(|H&LxG$O>&C5Njmh`31t$j*fY`bj%w8)Lf^5>it;VQUa>x?jWoz zkcHkG4K%JyJ*Jl~@7i1&dC{K{+9)oTjpI(7z%Bl3edS@|kaSzCqjUfc#|2Ds9Jg>1 z9Raf;Q1a@(*+*ITtOr6IlnfgiJL0I;S;!(^zH!;RK%iXcw&2)@!F+6dCGKLN1_Rp0 z)(F8Bakz3~>BkH^KUc2X7|vYV9EL`f|M5m}S1{BMv_HM#At;D>67YLW!g{n(L{4Jq zP}f}j!~MO15j6(jZ1Or(UAJnT|3tFeu*DA&ZeKj>L7#1R49o*^WZlrnTi(5(v?4fk z9UQ*B1d{#>@YaF+yThA8!vEhTrCfi~;}hEuivhckZYcTU<(AV)kI zN$?m8+7vwrUx;*eQY_}@YpBG_ZHQWjrMd!mSAs4Ky>A9%>pi;HX`z&Shz9*GFgoF9 zi5-w$)tbBi;N`Bq(ps;JoqP=ws|bSW+h<>(zyCA0A|n92Y55;=@(Z`Si}bjJ z2fw9d9_>5%9vYoK3eRHCo((9#pM$q~j^3T;=S(k621DJD1^_JeXL- zeb|8R51dU8(k8xI4{)*Y+MicFS7LyB4f}1(c zBv2T3VMVrh>CLIBDw1lCdBb9yn<^ya&hhsaGr_})fV zQfX2O6M^PDs1}Zx6O}z@YpbKFIIbCxS-;G(cJorxb=2aVIw+jj4AT{O{xTf&jx z{gmcwNdzxvcM~EMS=#F#PN_;#>-*dqf8hzDF;V{@xIr(LtmYD?QEe{@LJ8Yb_NM_fn)+mSDFVLg1nUr)F*&HNz;( z69rwOkqx5uCASI1YD+->gbvbXg%>`T3fsax+WT=2LfG+EZjK}=B;#LAebx%i% zF)MQ4>C)?J@=IzyYZctRmfl7Zjl-Q6S7IqAjm4#CYd_;~vbrUcpO_d&2#C2kkrhjP zZ@;bHg%OQfL@LyDt<*Y&gdz!s0zJ2(L(wb4@sUUa-TxAxaHrPvzci+;!Zo`^m(WP^Xb&!KaWsDuUJ}*U2PT#E{M5@_Re8uOTF@!wj(rBbQNG+;KC3H%7tY?j zb@$m5X|*_hJ1xPl#2jYgimID(%%s4WW@Z@_lTc1paW;zIrhG3;sfurQzIaT$ zg2DOtp%AE+yJYFd(|+P&y$fqcJ^O-T&CW0Cz+sS4xR);0n)eL2diFQVIL-yR6i~i2U0hVRo(f39Nv4q7dOL z8bp>n_#B5zQ7C6kA^W5;=YraQ{`K*zpXPxAWBg~_DpBXffnT>0*te#Pw1NGilcD%{KSL$p+)J_o$igc z*gle137M@C1pb28848Gen%82lVE4Z+N0xq^3gr}UHB3pE;5bFsKNClKHzJ}^3fAet zYZa02q|^zg!81Ob$k;QX3bXAvLOCgYK9X>VGuP{w-E=v%5qKo`U z2UmEz+RYIsYq6XQe9P%~+yDZ6%)#j+Hb#^_WYFf@6wb>>vQyWO7xYh2Z=GR1> ziYxSNopVc(1jMZJ;vZ-!jY&Kmr5Q7j4_Fa|Yp(5*_zTNvc=lGq`;tSJ#Fxd38A}>o zoUx_#S*D2UcOxTlT{=7F+KWI1% z-{zWNRm-5{tXiP`kZMsTDoz8}=T`PV-If%R*zyps`RmCJn_{=4Yt`kf-nW|+hj#;XcD+{rG z)Ng=_Q+Nx*{?*M6hlc;REH4!*w33qRk!zycZGC-+Ho&9t{1cbQP=+!rYwLGM^BTkp_W#M50ZX+>xkb7<(HuNg8s-G)~Z!C4QLb-r0nj2*~*7VUXMv^wozi5Ssa|fCt+5!RS23Z9&8gvpmaT^ zy-+DZgl1PnWpM=gO0G(%k}NnH)=v6glzH7kl6Vl6I^1nXpL2J`j*sTWOw#11&(L}_ zHf+?>vvas}1DLx>I7-aGk`~FdemB^9pIU^1R-PM%SOe3A-9|rq?Dj(sbGK>P<5xEbRwuT8O9?v z{FIQ@zs;f_u}nyd!_~v@vbtzd`+=3hj>lq7JvS@n>zz^ymZ&U}iJ-tc36H%5s}ZS* zezDk)+l4K3-CEAo`AumzQK^O+zQ}S=C1C34S0&mLZnP}DEl3aE_;U}Jn}V6FTh?mk zLx%{*rs^3uUUttuI#Iu-e0J?z0yFg(#&{9JUaT$Z*$p!ff4($`WOmz?ul9~mse-Nxa#~-J@3;daqHh@NE+PTkD9FFP-N*`Gpz|7OSr@Fr9bn5$D1Euo5m{lspw5w zKg!Nrl4Sm}OYuyRYD|ZRvI0EQ@$B&gVnL)6wTJ>BnFW4)c#jNl6&q7gygsd3eVV{F z;cn1kgm5Af!Bp)!-;l41y=rqw(6Jltn_;InBj_dMK@_tVXcJ;U#5zPgw!m)MajNeF z%zu93B|>r*)S!?&ox|I`h&baTUvk@trF)LDqobzyh2_Vb@?LHm2hnVug#u97!7Ky8 zNR^Mr9xs^6?>BVIl8>EVkeafd;#sjgA;C&&e>obQwwYE;2s-Xnz zF}>or+Dq||{_+3(97QhAoBs5TvN)}Om#0>5elepw%ym_|v76eTcpt{!C&s5fiSTr!Y|{$?SK!8QN>+96Ny+OKKhvke<#a}K+L)z$ zwm)|T!`j`2t1@RHZAGhMvL}+`9TlEz0L#$bk2y?z?Hc4Y)32sG{WkTy5|)Excuk`F z)G`7QCVCiuADw@@FRu0&2yQO87wWrMfRaNmOzU#8iz?{wK9fVy9mlnLRrEN#>v4L; z=WgG9d5hf3;X?5WrH5N4`>U7~7GD7r#wZN5_Ct!&{kU*iSjMB$WonjpyKQ;7fZfaG zYri4J&{1A%vIlWyZX}vV;oSZ065dLACpaYtVcUpYa{DdA{^q65?gU3qMdM(l*1xX7RH|3}g zQa887$cZFkEr)op9AnC4Q*>gC0931JK_mJ|SWe-yt~; z29mvfZQIxo)U&vT{`-=1`138q1Y98eP*=ckbaozCT6}EzR%zraX2O$4A)MYHl(vF= z)N%F|^fUb&Mo<_hP;*fH{S`DE+hKXwg5@9JFVuO~YFzt^-9`KCEl{|X-cz9)K(-&< zcXb)vC5CohXd|xA8U{?6vJy6Xi5et~03e(*&bhc%FyMDXe+FFs4i0iJD|>MKnFwd8 zQHI`bH}Z}-;7Y)~vnzZ4zWl8R^=ANtf3imK*U{<^fLz5jj0r{C1rch~+kzo%7qv`C z1@4qhTy_bPfAM9zwRC=?%WQz6YhK{{Yr;gEU=XWXUPWczi7Aahn)lk);qCXQXAz!94crZ zpDWDtsy;t`Hqp^u6TOXn4JEutiHksXH^uHJ%tLbr-&sy`7u4+qXu?fEg~T zosD2GNc|QU#B1#I-RuT0n5pITy2+Hg*~lm$EeqPX!YJGgyJhP%ZYgl^oHTWqUPwdK z5f9iSR2GGJi&Zjo02CGYU8g{gIq(i$%u`z zUsOB-c~H_%Jy@Rb8r&p1oJ-M(jqS8g0jhUHv){(q@2@(~J4=n8?i}d#Mw>`CxU>F9 zbI>;;A82cp7&g66k-k3qu#+{E0JW6Uh@wEC0URI4##ygchl%~fQU+!Kx*n;{3pui^ z~64~y5eJwZDSZTdouKhm50ltb81e3_Ny^?EK#@;WL!MHHa6lnJ4`l6lxjea}Du^IDo zYaNTT@voiMhK3gHvW;GFn*#AJHt#Q+ga%ot!1sXby*U@-v51wS*;W6uXuxwZu?-#O z;whf@mGUpD=k$=_uea4EVu7GjYQ*(T5y?uA8Pe9Z<^72hqYd*N>+UKRu2l?Aj`}ZM z#d2s*uAeIq;~PJve=FA&;nz$J(|v38m`3a# zV7}?{WFrbUT^0ei5J6!&Pz*Oc_Gbb63G(AtG^A<{9nxAOzF&_XpfU}G!U)gSGC)(P zzs0bR9i5F!R|Z81Yr+a=YUd(h^4z>7&$4G7O)upb#REwE)C!l1mgS5L_19W58NaT3 z)V>244{Q1q;P1D8{S@_(g`;;E1X?+%IZuhGKQ81pTMi;JA`O9-x)7fR#L2>EKvKAC zpE07AF;_i|CJu*b?6VHjCsdgGJ~r4SY2fX8weIcmY5#+_3;qa-5|RA6-GuWG6R4-Y z@(rf;bY8kR&OA_Wya$GS!kcPGeuRgF3hU=nRkAT%H-M z=K1_WZQ1UIGkfNk9z{WmI~M}& zQKu)fGx@q3U@}kjUiCJ9(}rc~_KUR%QF@gxlU+KwO?BPcQ|*1uwl}QMxHQ}hKK15l zu5x+Wq?XyZ3}rrnnOHGBton(KYOxp9D2%2%yW)N})*vi#MWnp-AIw<3%JQTqIez%HIX z_$xqul&%oTerF4YM>CCb%X)n&M0-_kNBLlXc-ML{m+{rA%Tn#EEsvMvGdiDTGNVsp z`nD*K%czp2Y~34zg5Bb%Ba*f&*7eESW1pT=cuh4RqUBd^Pu2v#>F+sW{p2aOxFAjU zC@d@8w(LtS-5AYv-`vWfv=@wSx^^UY|lC+56-(!_-U~OCf&5;Y%J=lS11sWH*k*vQl?*2-Ky4X z;)?z4$J@>t2qI4~7KD4Znaseiek{*`Cv39kiH=XT{2_88qyXjD%h~?*{kf>A1fDgb z2^ZMHQb!g{SDch=VgR8?-8ff>xHUGDN-V=0q6J(n=3){wZEf7$hhPF!=59F`1*yBW zVYH&lPQ`JP;8)!|rGU-E{ts#RZ-VPl9=fvSjaE|i~6;`yi&c|KutCS{Mw6*^2xu<6h9 zTv~gM$TgNz3^^LC-Cbw@Mw&CGuaVMRjRnm&x5eBb#23NFT$YJHZ1|l1n1UzB%s+8% z@bO>pssr;vHscN2$If>!Ykd-Dbq_JSojdP*b4jvlz7tkJYUWTJcfFJ7w!zCUt>G;F zT>*{UI!KOw%wXH;Y=x;|M_h)9O_BcHcW zO21G&UndfES$c*{FQ3Hd6FuHpv2wERi!?~@G7~ZFgrS>}4=g_Si68D;sP+#xI-}TV z=yoo-<+{GZW+8YaMwPqdgC>H;q@qob(CiOnD}g~LSEhBrh#$-l&j>WH|5+Dy*)5x z%27G_gf8}e&^kGgvO{^s_p%_LTu1W2^qZ<(%7MY9@Jb7g?@+n23*%#pdT>s8RW05i zEyZ-b5jRNF9;`j>ZRa>+Cyw+9QM>iJ;-QH^{h0i(vd89{$vxOdnm9LArZ6;jU?`7rketFy^NWJY z%*Vx#r9JX*`dhn`?>a<$Ym=cMV&?Q`O8k(B46?KpEKmd2fO-CvIV|(gXcY@DIX|46 z{^J>X{@kjn)oLnfvLk4M22d(B4)m(sLbr6Yshq^qObJR`(?}2=$}#$E-Il2vG$Q~5 zktCIf+(`g zZhcNHYVnj>Kbh;KjgtypI{`2E3)8-1<)uR>#}H{?F&$IkMZ$`^t|o4hwfHf;Ui~({ zb?}_JKW2cLFRSRqRjLU`%E`k_!W7AuFe-DcA3ZVV@m|bb9OWQ6?2-2xPj7I>gG^|k zFE*S;m_aYZ+*j|eV(~RlN;j8=63=MRHt$qOC?0#PTgSWGa>^(kwXP(;_js}T%jX1z zEN)-1Wh#FHix#a2%=GD}8#z9&B*|D%MZg@YIQ{U=O2mZ6lb$2WDKgP$1O&wPzafFC z#yw-B$jF9{fPd^Y-jzX2uSe4420?3?K{#BI0uLl}mQcZ{MfyBJA%-CVD>R?fJT^df|orn%4_*OP)v(kGc63oiGS;*=ZPulkfk$)*6c~ zk-S~V+|Nbx%v!wEIkWmSvX!)(&iZv#r^-OIw9`jXzgqJgn`Est&G%Ew;yn}Qb~h)Q zC;rw-y1-Uxld|IhFs6fza5c3Jx%bYEW`V7Lf9zoO^P0a@`n+U&oX-sl=yxAkT57hgGa_5 zQ+r^@`lcf{Ux&D+8}QZ*braRw%oJ4@d*vEPbj% zbly7CACf5apThGP3rKt(@m{rTz|y9bfw)#M*Ek4-6y~la^=3LewMmM_Dc$$WsMX8V zV;mNji{&X(M)VwbRM%`pSN~p%FxMb0efVH8xBA1BP&p1$t) zOEh_E<5)aG8Rg@GdS~eTAx(@51up7P?(00&b=VifW#s$6+7LC0M)a%QNOYzmm(aGq zp|hm2ek(5|x?iduyZE_>f}|Q%p8D#QkkH6P+XSAtYNA2zvR(TLcK&LUlA$ROb~e;4 zl3tzD=;o{6*lg3Pm{^VchfkVjR&vv)hez!Aia*3C|Ln_3y_s~+SzBrPRoiLI>VEuz zvZV|1U-@{JpAwrZ!Xk{(cL%VV3k2f?QWpv^;+!vGim={?7a{61_3I)-NC|jP;+}Yz8Qv2`rt-NHMGSoyy&3x1j@X$gd zO*Wf-6koXJCH~rOzrfvpi`=O5s96Dmipto{?*_-IKG&O)cQ|8&#^|Q1y$)AhD=5vh zIJlb?98L{Pei z)232PUxS30OO*hH(Iw^Zr&ws>VgsbBF#%@lrMv<;VAKLBE7_B)U2}7jhpn~O%2MTb zSRIM+K{oDVxaFQ~TE>t$wu2&dPc|(DVBRSz%tqVSU!0;|22gFS#*ocB?`AYy-8A2E zF>UPU56{we?O;0|1Caz@Y!T(Kw`E+;poJ z!0ubUj*9;!l(wffb%^__4~zcJdkk)7#s61$&pKW>CIWX`{jFnWa*1-x$Yb&$K zuafFv7>&i*)kFg=fePtk52RiXLiI}82_V*gHf>kdzteNQ1@6zM@?ScPfBDP*Ab;70 z1snH8HlwM7l z7^5O}m5!VSKbW%AfwrT;YG<_X#p;rV-hDGw#&K&M&IAG$ZMeCSY-4%Vr0EW!VB?Y^ zynhc=NnKBjT>>SLa*q9&>#lrZ^;BP8&vkTzHnTnBFbS9oNbRl-u5dq`U{$hs-Y;qfzwJqF6-?ECMcaG2X;)^WN@iIUQvbE}~duS5k>`PY`yHN!477v14fOP*= zp+l*fCM=-v-w=C_z?Eq68#Y@&{9rn`X~WCq7cjZ3czU8*>(S^?scxK%ImfCSWD}IH zTwTug70&#-QeA$s5%^dZbsY{9E3p0gHNb8gXc_mvo!yp^w>G9s>VQwRv5>@0i@Q%Q z*tonA`{8`AHT_TST48`+k0#fm{PlnTR1UkLyY}B4&?U6E`Q#FCKVSXzkImqoy*ehl zr;ipBe4s98xL;Hbu-a1O~-v`b*m_>VQl0m{?Gz&%i;^ZQQ;Fk^F0$hKYzGxUMS>!<>-FraXX_BxK{9Pv zT>ow=8?xC5v)5{)5A!;zl~N7)DI=Sq>O-l7o?#l;?Ka4Z6<>7@^9y}u(hAUeuW;!S z4MM!T0Ep3hW4d)TaPkMp5|5IyZ~e0+#`HqAaB;^vN^-A_gzdYt00>uc=q4H@d7}tW z`QJjxE$DxuaqS@--+(vo;NtbLpGq*!HL||{Feo|n zA%Q@k+iTdgE`dIJ5TdxmF!t#tHZ+%KJ5XmeqEz2?xnXaAWvc&FM3oSyd()_n_ zJyaLoKZ!l*sER7p=YD26C+h|eNjcc!Kk~}U910do4#hvy2GvtQvX`!MZZinJ6*S%h z0|B=(WNkhC=tgL+t!dLeT~ymQ>qQP@ z|G38as83HPD#T7!9!m~Qul#dl@VV7uDl(DJon2#4PvQQXv(baijMg4Jsk`B&E*JVS zE-jVo{p&50L$x1ns5KN}JgSV!HBy$?dh$jgHyAT?YfqE(x6Wy2%(W@Kj^xmWT72x} z^e@XkJ?HE7vCyC@<%!S$9#H(#4vPjVwog5V_|Poz$cYC21^$&bIPT%PQ4&eZ=w<=H zYj&SDu2K?IUteZ#t60FplH(6#XgQ!tQU_w+p0I^FK55N)_thx_&&;lKm&<>G;@eV3 zK8KZlO`d44)OMcxF^*^U=^jhHw$JyN=;hbDHgELUqwwzCVMD{~zI@gtx~hnGhKA)= z(nCz#m@l3Ud6}Sng_9#l^*!U;O?mWQ`*jyU1Pe|HLtCErsff%T%SHW-G*s z9nwPcwD7HHIDn!)L4uaCsSkm@%5u6ib^o8%2B0#5l7nKkZK(5bD8Qt!yCz(~1laZ` zefiG~dcR8^#tJ~89NWCAHaf+0E1nC6T>h_Aul~?Ko5FR7df=Y~D8JoL%UC-Q|CaGE z>R_2}-=T_ys{B2?vSuFdY65l~22#Xjsf1b^RV!#f0$Yb=<|Tw=5aRDGSmVpf8wrVkD*1n41uCC+%iJT&x#D#8~e_TpHj-!iV9jLs=w(A?o{3V;sTh=~5dH%J>_N;7A|M>p;wAolnkb&aVHOp&r`nbzMA4KA{HN#F9(ZsM;ax0haK z&{5!aWi)!1c@bS-gU?`wtcB^>CIls96``P z6KsE&@|7tus1O5r?-DiPFi6*7wCF0qcjvUWyrJr4w}AzGclYIKDu_k5K;M0?5(^0n z!kb|`g%f&t2)+4JW^X zNd$vNP)$_E%iCqwI^ept##!Y~Kk`A(W6AlTt_hQajrEzpoQ86l zPqpV>Wu>cm{&=C`MGXf*#@vCN2ED?ZCe3u%vM;#HtSNeF)lP|Hs<4xl7T{*P$SCBQ zjB)8H<6}|N7aMvZRT6Uv2;m@#BZTDD6hf3q90g^ni4y5GiIcAaCrjWOZ*^~yR`j)~ zFXr`zPp?5<)G)Z0%=0n4%G9FV)^r!&^?+NKom_$Xm|GE)e|wGyGp-4+D9Y*#K2N5F z)S2dx9xsiMF^J@ho1gW&odb$h_EBUamPo*;CcdY}E zn)vySeAv3x0TQ|xvGpx!KeVCf6Y%`^y6}})XRW6gL+jzBBI-@kFUb7`3R~-WIZR8O zT>>Z(uYR%dRR)!6Ucozse1nKba4-H<{{ozru@`|2d~KcQx>jJsYPe$@fU1N0zoDtj z^bs)jqIzfzEvT4+d;kaSX)3>BSy@vpjh^WOD3KiZ=4*o<-R(`>_SW)ZRzgRhZ33d5 z-Z~c+xhqxVfpm(b2?=x{9bxeDX`2_k()RvJvZ)Z`RdU^usbjtKb#kvat=QF1;5s61 zQRW049h{ll?ucx5%BCi=v-os-!r=V3)VdhcG^)NKgU>f=&l{g?e9LC3f<81Tl)Wh(>68X?1! z6IgvUtQlT!d0BQk9?jeQA#5Rgr~M^nmvH1JHOBS&c&yF!s)2o0sMT-F4r zzojul%i-(u1OUW+jijrzFmDocbaCpCCoN;uE&w1D4>hN*umnV=wZ)@PB+pBBbbbE;eT1J!vFRIKNGCuLPBw{1j6Y zUT_3yYTsU#ww1i2`MT+oGdxEot{3d<FU{&^=e;A>bKa20NdP_A zd?$Tne}|A^j+Bjm&k6bi2r<~(W*KEWi;x?7o7mH38t}rn*<9*710AYS^8oO!IY|EKOkc(B|A$!FM}FU;F78~371l5qjjcBC^|w`z$> zxb9sOB>k>)Q52@vWbb-{(8%F7a#{hU7^I#Ca=3{*0z9i@GT5FX5;$vC+@my_)hXdV z`ve)UWKfq>b`Dp$gyihuNkjb^t>2e}evosbuby?}C?Oy!{jGhHi@{ssNh#Ur(pv88 z>rAty8d3v6oyFPgQx5lRrF@cQmfY&bc~6;8_UuQ(7%P+4d}it}Y%pG*Hzb&yrF+cMsWwQTURar)4NX?C_n zcGk7*ASICoJTE_wv_jBUl_>e`kBu$6CdFGO7rf5jF<$~~N6dCP03qJi$Yg8Fae~mH z8gYV9BD%XXxyy!{qhV zg7qB|R<#+|RXyUj^`R5`iLkst-!QU3s@yYJ(ujNH%uzt(;@$U>U4~oyUXVVmis29t z?&V+{{0cYSi51C;Htb%QYhWyCw_}|XOkb#7C>dPTe#E>7ehhw~&P(6*a2cUpLbiGF z^!#5nMTCH9hX=F2D>i3nywo0*tZ=R~-j2(B3P{zmXlK0?s`A0Ze&O_FxQ}}hZZJ*z zbCB{$vlo4h{Um+(JOdAlQh08~LOnOLKWaGJjhTT8IyK4u3zhr^90@Dh*Z) z5iQ{A7uk4k`0jR1(h4gI2s(zI4XlWP! zbz%cbX-|{Qfo+=duT>~H5>nUqj0A=i6dfc$vKo1VpntLg_O!WRCuMa zLYHTGjWPFNkpImqiUP6&t<8F3XR$m{dvTm2UrDn_Pqzw}=}*W`d48-&XeV0O#yH5~ z%V!jz?QIH=eCS9OpIx4+fMod4xo`X}z? zhrYh+xKtS3#*_Lj+-Dv;b2rEADr&qVt;43n`*hN~$@)O?C4>6xY`L9A9h0NK+j{C0 zWBJ-md)P+8Od1Gy-vrUxGRd@8@J-d;Y*{mnXf;mjC9h8&#MhYPeg?deA0?Ybl6yJ*|@GoUXHQMb?qx11EdNO6+gVUZNj0tqk%U@XV^P>N(nkeFx%K{iqL_C z)BEIQ9LTmIU19OGY-mvQO}#l>S;Sk-z!n$&=CgTvdY^eDi%!ju5|N=w1B7XR{%9^x znL~%)UEkSj=>%Vy=cK)yNL#dB<~e}@*U$xqnGO8jl6`qd+y3{gBFLNS#WJitTxlJ6 z)HOelY%I>l<}Ai6P&hej{P5$T9qZU`F+Dz%H{Mg-qrl5McY4xBPB%RoZT|qT=>)%6 zY#2^>6`Iyj3G;#%izp4SSvE}F_+IgV3sgKA!#N?$DzhZd{TQo` zsuYh+SM!>G7T2)wY7G70qepI@AlLk0eZ9iWezpLZsH^Gl@+C*W;YadlEWVyiL}YcCD&MQ@kDT8H}Am&;yUihIJmyokebN5rteQv$>EdDw% zv?M{abS|w`Jb9=u3E34gQ@+oSMXpGqB7K{th^-Au;c58+o&vx ze%-@+=TJ`aVLOA^(OuOOB6_Aho#sh1pxSxaG`!8q@@zy0|Dt5yVievz^Hio3 zs|B!=d_hD%q}LNIRHgpQ^wANt)sa&T1?YoD;_Iyf2DjAzKJ(el^n_%+B;H1Pi%8U$ z+0EqrL!7Ya;K+00vnXsb_>0FM2aL=EZrOL`#AgeRzbyt7cED$&JPXlJ(mFbfMaoWD zSG4#!9n4(0$1pif=f6}g^Ww!GsQSv*qo9UDZqnra8hul=S9>Y1VkG;717G*qIB=MF_9|gGrDO_oSTJE*+Ddbyvu!VippwD%!?Ur-d?mA}#1PNoQe4nY++P}?gu=w#bU`%ScJsIppP0vR z8OO$ONrxcmQ++S*+I3~i?{*N||Jo47Erbh_mgswLcXeAzTTr}WC(lGpT#;H|c~%Yb zrk9T7K&6z*;9Mr59C_}Z?eASbG%{ifQ;xA3C9M~tY+^=xCqCS@(+h5ih*$Yj1$HqGPNynRI`D*fSH3VhwPUMzYj+LS z^IB5&u2N!z_DlfMk3B0>iPuV3HU(wh98_#`-P5uqQy0bU8O{C}+h^<=SfL5?_9&K8 zHuLejf|u7wM$Q@(Px38_{grM==i!|ss~b8k$(AOjT?duwvt!*&B! zURlc0?65-RIJt9A?`2(r_06$5JiO_mh^kb-ThV?8$;go<%%0g#tn0gX5ZAMF5M`BR z-qE5`AlIkKuUg3#`z@ZIr@^-B&b=At!*+bth2eIUMq+cHThkwkA92{yF=xnqOTT+O zJLgnS=#Fj98qQxQv42=bCE)Ai!dqqz_MlJjWn)`Pwx1KYZvL_c&M$WTE$LRk#Y%SO zlSk(9(eSB5wJ5JAPoRX>4nENtg80}oOYguIMuO0w%X^Y)n|VOP4D8QB5k1=sVF!YK zXEYA$h(6DmEk>m-TImq5rC3{*d$BA4BVR6>$fF+_jbaYH+~*U93ev4Tl)5m9ECiW; zs3Q+Q%)EP7_IC36CrQxP4`;uHq&N!9l2z28ziE6Tzu4o?>Btr(vzO`}DX53j{(SEG zd77eacE%l2B95V7{*pS;HM?zMzANh3V3q;Uh#!$|Y}xqdZXv`H4fKwQ7-Xo*g6kb~ zIR+S8R|RgHry(r8Wy2`&lMo1o(nasMLj0YYiTS$4h3>_Z#vyu!!`>yNieYiHaD6Ec zw3xbfEal!DqX;&WYG!;}ZiA!{`uW+rt{rI$ntf_9g760WrbcU4A54b5X?D+wZVc%?zu~gYC%C_^ueRuhH2jUlP9LjakWesC;aj_(JnUXqdh~|mdvU~Cdt+TU3Bo&Sdjvfo=bIOYze`*TbkA8*|D;F9> zey>H&*N(h+Xnt`31+N(FK4i3U=|;c|tUG|u=d))JvV&cDI{alQU@XVYs>(YU~2f$QcUf!(eusfoizqJ9| z=J>&W%eZbHCn*^>y_nF`+_a*f0SAOJz4KAND)2!9YGX2bwijoLGk%bxUo7q}gD+1b zHL-3xI!fq0?0HsavHP!?nb3e*!`7L;o2YW(eBquwa$ee&@#3Z=hH9da+~BJwOR2jK zN9;yl9OY$3Rra-t?HBY}e>0_1y3QG{;~>Ch?p#t=Ijz3oOctTYs-M7*FPKe^E1Vfb zm8g0jIsR1LJ6X81&IYC@5NZsYaE|xssSh^etJ^UC&JD*fClSspsfo-v!iN?(x~zX+ zvjE3uc5}Y5OwQHpjRc|9(I;9_AcW?El)0>$FXkg%%EOr=!9GvE?!1>{YIU>YNC$sV zecgtC@mSw$R%0F~)nc5o0c%=L!Ldz62m2b3YQ7&8eN|K8WkqvL$n)cC31)vN!3?q4{5%}Ti=ViM z+2VJx`mj&fumU0u4zl{Vc#RFn8?z<$IO4}9%mTcK0Z27f>rkmu)j)HnJ&A=&ySJc) z`xyE0%rG>YlSXm2ve~uKZAz{vvz-nqhCG~aPl@qiWvb6rq7*+f@_}YNRmcTS>5uqG z>Dj;H{zwUEA&9&ll~pXrEmeswEXhelaS>nYe10FOKOsNgD6o{_)AmF& zELW_rku}#r3dV)=Il$+G$h6HpDrxrDZ=RiH*G@gP61#q=GMXnXXTf9VK$f4%J9rf# zTcrR7>Y^ov94amCT$0iq`_$%sCc`Hpp|0(n^K_!=h+nvAMq$7SYqf6%FQy$Q=!ewdRjOw?u@3x~ggHt|I~}rb zcE3@z)j!fZpBh}c5z^}k3P1L5ny113&6}ra_CR@5b7uLr<~McQM<+w)0HF_Y3mU9nU|wRROZs3J`=CrH(fBdcXSfrL8{w5I6&8 zw;`=MUu9Va^2p(KWef&okxbBcw7@zJzdpQjCMD_u4$j5p*KbBCsPtN8c-zNUeGgA|ggb79 zoH~VSx@1#{+UO#1_iQ2!PAWdzHZ2PlxT?AWUT@Nf3S|3Y*n3&0U%jv;=`@*b^f1#e zhT0l$K>u*u>+*fBv*6*MF9UwA22%yj_^TG^MZM_uBDf-^M`X<}kuObrM{c{2&m*nb z9rq@B%uGR1WgnPxV4E!yp&HLW+|4F%_wd?%*!7p`0Tg78*#`V);)~eix-oF!oK9?B3<=?=o;}0rdr{ zBFp<=m%qFw+vLJzD+C}dFP_ZcB zz}*9qKFwA+`&qUHu!J_WRHlnzD|4`wublR# zl9vKP7Qp*HMwe$MFY1qDlhV03j*Yc^lufA#m4%{fG7a0NS0ei`zI4$F;`@@{FkcXG zJU2xc-UTmm4;`!A>SmyKj~u=8X%1=)a%}IgS8j4=*X#)Ts5&FQ4*WT* Mq@kF6+T@S_0)*!gaR2}S literal 0 HcmV?d00001 diff --git a/src/current/v24.3/alter-database.md b/src/current/v24.3/alter-database.md index 5f09b5d0cee..3ceb1e17137 100644 --- a/src/current/v24.3/alter-database.md +++ b/src/current/v24.3/alter-database.md @@ -169,6 +169,10 @@ For usage, see [Synopsis](#synopsis). If you directly change a database's zone configuration with `ALTER DATABASE ... CONFIGURE ZONE`, CockroachDB will block all [`ALTER DATABASE ... SET PRIMARY REGION`](#set-primary-region) statements on the database. {{site.data.alerts.end}} +{{site.data.alerts.callout_danger}} +{% include {{ page.version.version }}/zone-configs/avoid-manual-zone-configs.md %} +{{site.data.alerts.end}} + You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. For examples, see [Replication Controls](#configure-replication-zones). @@ -689,6 +693,10 @@ HINT: you must first drop super region usa before you can drop the region us-wes ### Configure replication zones +{{site.data.alerts.callout_danger}} +{% include {{ page.version.version }}/zone-configs/avoid-manual-zone-configs.md %} +{{site.data.alerts.end}} + {% include {{ page.version.version }}/sql/movr-statements-geo-partitioned-replicas.md %} #### Create a replication zone for a database @@ -715,6 +723,10 @@ You cannot `DISCARD` any zone configurations on multi-region tables, indexes, or ALTER DATABASE movr CONFIGURE ZONE DISCARD; ~~~ +### Troubleshoot replication zones + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Use Zone Config Extensions The following examples show: @@ -1078,6 +1090,12 @@ When you discard a zone configuration, the objects it was applied to will then i However, this statement will not remove any configuration created by the [multi-region abstractions]({% link {{ page.version.version }}/multiregion-overview.md %}). {{site.data.alerts.end}} +#### Troubleshoot Zone Config Extensions + +The process for troubleshooting Zone Config Extensions is the same as troubleshooting any other changes to zone configs. + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Change database owner {% include {{page.version.version}}/sql/movr-statements.md %} @@ -1283,3 +1301,4 @@ For more information about the region survival goal, see [Surviving region failu - [`ALTER TABLE`]({% link {{ page.version.version }}/alter-table.md %}) - [Online Schema Changes]({% link {{ page.version.version }}/online-schema-changes.md %}) - [SQL Statements]({% link {{ page.version.version }}/sql-statements.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/alter-index.md b/src/current/v24.3/alter-index.md index ff2106056e7..eee73759712 100644 --- a/src/current/v24.3/alter-index.md +++ b/src/current/v24.3/alter-index.md @@ -47,12 +47,12 @@ Subcommand | Description | `ALTER INDEX ... CONFIGURE ZONE` is used to add, modify, reset, or remove replication zones for an index. To view details about existing replication zones, use [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). For more information about replication zones, see [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}). - - You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. For examples, see [Replication Controls](#configure-replication-zones). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + #### Required privileges The user must be a member of the [`admin` role]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or have been granted [`CREATE`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) or [`ZONECONFIG`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) privileges. To configure [`system` objects]({% link {{ page.version.version }}/configure-replication-zones.md %}#for-system-data), the user must be a member of the `admin` role. @@ -225,6 +225,10 @@ You cannot `DISCARD` any zone configurations on multi-region tables, indexes, or ALTER INDEX vehicles@vehicles_auto_index_fk_city_ref_users CONFIGURE ZONE DISCARD; ~~~ +#### Troubleshoot replication zones + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + ### Define partitions #### Define a list partition on an index diff --git a/src/current/v24.3/alter-partition.md b/src/current/v24.3/alter-partition.md index d8b0c1b3479..93035174565 100644 --- a/src/current/v24.3/alter-partition.md +++ b/src/current/v24.3/alter-partition.md @@ -9,6 +9,8 @@ docs_area: reference.sql To view details about existing replication zones, use [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). For more information about replication zones, see [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. @@ -44,3 +46,11 @@ The user must have the [`CREATE`]({% link {{ page.version.version }}/grant.md %} ### Create a replication zone for a partition {% include {{ page.version.version }}/zone-configs/create-a-replication-zone-for-a-table-partition.md hide-enterprise-warning="true" %} + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + +## See also + +- [Table partitioning]({% link {{page.version.version}}/partitioning.md %}) +- [`SHOW PARTITIONS`]({% link {{page.version.version}}/show-partitions.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/alter-range.md b/src/current/v24.3/alter-range.md index 051468f3da9..485deb4d4ca 100644 --- a/src/current/v24.3/alter-range.md +++ b/src/current/v24.3/alter-range.md @@ -34,9 +34,11 @@ Additional parameters are documented for the respective [subcommands](#subcomman ### `CONFIGURE ZONE` -`ALTER RANGE ... CONFIGURE ZONE` is used to add, modify, reset, or remove replication zones for a range. To view details about existing replication zones, see [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). +`ALTER RANGE ... CONFIGURE ZONE` is used to add, modify, reset, or remove [replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %}) for a range. To view details about existing replication zones, see [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). -You can use *replication zones* to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. +You can use replication zones to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. + +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} #### Required privileges @@ -121,7 +123,7 @@ For example, to get all range IDs, leaseholder store IDs, and leaseholder locali {% include_cached copy-clipboard.html %} ~~~ sql -WITH user_info AS (SHOW RANGES FROM TABLE users) SELECT range_id, lease_holder, lease_holder_locality FROM user_info; +WITH user_info AS (SHOW RANGES FROM TABLE users WITH DETAILS) SELECT range_id, lease_holder, lease_holder_locality FROM user_info; ~~~ ~~~ @@ -163,7 +165,7 @@ To move the leases for all data in the [`movr.users`]({% link {{ page.version.ve {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE LEASE TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users' +ALTER RANGE RELOCATE LEASE TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -205,7 +207,7 @@ To move the replicas for all data in the [`movr.users`]({% link {{ page.version. {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE FROM 2 TO 7 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE FROM 2 TO 7 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -231,7 +233,7 @@ To move all of a range's voting replicas from one store to another store: {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE VOTERS FROM 7 TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE VOTERS FROM 7 TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ @@ -261,7 +263,7 @@ This statement will only have an effect on clusters that have non-voting replica {% include_cached copy-clipboard.html %} ~~~ sql -ALTER RANGE RELOCATE NONVOTERS FROM 7 TO 2 FOR SELECT range_id from crdb_internal.ranges where table_name = 'users'; +ALTER RANGE RELOCATE NONVOTERS FROM 7 TO 2 FOR SELECT range_id from [SHOW RANGES FROM TABLE users WITH DETAILS]; ~~~ ~~~ diff --git a/src/current/v24.3/alter-table.md b/src/current/v24.3/alter-table.md index dc183d5a518..3595a373921 100644 --- a/src/current/v24.3/alter-table.md +++ b/src/current/v24.3/alter-table.md @@ -223,6 +223,8 @@ You can use *replication zones* to control the number and location of replicas f For examples, see [Replication Controls](#configure-replication-zones). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + #### Required privileges The user must be a member of the [`admin` role]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or have been granted [`CREATE`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) or [`ZONECONFIG`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) privileges. To configure [`system` objects]({% link {{ page.version.version }}/configure-replication-zones.md %}#for-system-data), the user must be a member of the `admin` role. @@ -358,8 +360,6 @@ For usage, see [Synopsis](#synopsis). `ALTER TABLE ... PARTITION BY` is used to partition, re-partition, or un-partition a table. After defining partitions, [`CONFIGURE ZONE`](#configure-zone) is used to control the replication and placement of partitions. - - For examples, see [Define partitions](#define-partitions). #### Parameters diff --git a/src/current/v24.3/backup.md b/src/current/v24.3/backup.md index 7d5b004a881..8e6cd5dd042 100644 --- a/src/current/v24.3/backup.md +++ b/src/current/v24.3/backup.md @@ -33,10 +33,10 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO ## Considerations - [Full cluster backups](#back-up-a-cluster) include [license keys]({% link {{ page.version.version }}/licensing-faqs.md %}#set-a-license). When you [restore]({% link {{ page.version.version }}/restore.md %}) a full cluster backup that includes a license, the license is also restored. -- [Zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) present on the destination cluster prior to a restore will be **overwritten** during a [cluster restore]({% link {{ page.version.version }}/restore.md %}#full-cluster) with the zone configurations from the [backed up cluster](#back-up-a-cluster). If there were no customized zone configurations on the cluster when the backup was taken, then after the restore the destination cluster will use the zone configuration from the [`RANGE DEFAULT` configuration]({% link {{ page.version.version }}/configure-replication-zones.md %}#view-the-default-replication-zone). - You cannot restore a backup of a multi-region database into a single-region database. - Exclude a table's row data from a backup using the [`exclude_data_from_backup`]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#exclude-a-tables-data-from-backups) parameter. - `BACKUP` is a blocking statement. To run a backup job asynchronously, use the `DETACHED` option. See the [options](#options) below. +- {% include {{ page.version.version }}/backups/zone-configs-overwritten-during-restore.md %} ### Storage considerations @@ -378,3 +378,4 @@ To use an external connection URI to back up to cloud storage with an associated - [`CREATE SCHEDULE FOR BACKUP`]({% link {{ page.version.version }}/create-schedule-for-backup.md %}) - [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) - [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/cluster-api.md b/src/current/v24.3/cluster-api.md index 740e94f519a..e10016abd8a 100644 --- a/src/current/v24.3/cluster-api.md +++ b/src/current/v24.3/cluster-api.md @@ -21,7 +21,7 @@ Endpoint | Name | Description | Support [`/databases/{database}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseDetails) | Get database details | Get the descriptor ID of a specified database. | Stable [`/databases/{database}/grants`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseGrants) | List database grants | List all [privileges](security-reference/authorization.html#managing-privileges) granted to users for a specified database. | Stable [`/databases/{database}/tables`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/databaseTables) | List database tables | List all tables in a specified database. | Stable -[`/databases/{database}/tables/{table}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/tableDetails) | Get table details | Get details on a specified table, including schema, grants, indexes, range count, and zone configuration. | Stable +[`/databases/{database}/tables/{table}`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/tableDetails) | Get table details | Get details on a specified table, including schema, grants, indexes, range count, and [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). | Stable [`/events`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listEvents) | List events | List the latest [events](eventlog.html) on the cluster, in descending order. | Unstable [`/health`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/health) | Check node health | Determine if the node is running and ready to accept SQL connections. | Stable [`/nodes`](https://cockroachlabs.com/docs/api/cluster/v2.html#operation/listNodes) | List nodes | Get details on all nodes in the cluster, including node IDs, software versions, and hardware. | Stable diff --git a/src/current/v24.3/cluster-setup-troubleshooting.md b/src/current/v24.3/cluster-setup-troubleshooting.md index 80c3fa2b930..35dfd522e9f 100644 --- a/src/current/v24.3/cluster-setup-troubleshooting.md +++ b/src/current/v24.3/cluster-setup-troubleshooting.md @@ -587,6 +587,18 @@ If you still see under-replicated/unavailable ranges on the Cluster Overview pag 1. To view the **Range Report** for a range, click on the range number in the **Under-replicated (or slow)** table or **Unavailable** table. 1. On the Range Report page, scroll down to the **Simulated Allocator Output** section. The table contains an error message which explains the reason for the under-replicated range. Follow the guidance in the message to resolve the issue. If you need help understanding the error or the guidance, [file an issue]({% link {{ page.version.version }}/file-an-issue.md %}). Please be sure to include the full Range Report and error message when you submit the issue. +#### Check for under-replicated or unavailable data + +To see if any data is under-replicated or unavailable in your cluster, follow the steps described in [Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint). + +#### Check for replication zone constraint violations + +To see if any of your cluster's [data placement constraints]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-constraints) are being violated, follow the steps described in [Check for critical localities](#check-for-critical-localities). + +#### Check for critical localities + +To see which of your [localities]({% link {{ page.version.version }}/cockroach-start.md %}#locality) (if any) are critical, follow the steps described in the [Critical nodes endpoint documentation]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint). A locality is "critical" for a range if all of the nodes in that locality becoming [unreachable](#node-liveness-issues) would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. + ## Node liveness issues "Node liveness" refers to whether a node in your cluster has been determined to be "dead" or "alive" by the rest of the cluster. This is achieved using checks that ensure that each node connected to the cluster is updating its liveness record. This information is shared with the rest of the cluster using an internal gossip protocol. @@ -633,18 +645,6 @@ If your cluster is in a partially-available state due to a recent node or networ Even with `server.eventlog.enabled` set to `false`, notable log events are still sent to configured [log sinks]({% link {{ page.version.version }}/configure-logs.md %}#configure-log-sinks) as usual. -## Check for under-replicated or unavailable data - -To see if any data is under-replicated or unavailable in your cluster, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). - -## Check for replication zone constraint violations - -To see if any of your cluster's [data placement constraints]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-constraints) are being violated, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). - -## Check for critical localities - -To see which of your [localities]({% link {{ page.version.version }}/cockroach-start.md %}#locality) (if any) are critical, follow the steps described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). A locality is "critical" for a range if all of the nodes in that locality becoming [unreachable](#node-liveness-issues) would cause the range to become unavailable. In other words, the locality contains a majority of the range's replicas. - ## Something else? If we do not have a solution here, you can try using our other [support resources]({% link {{ page.version.version }}/support-resources.md %}), including: diff --git a/src/current/v24.3/common-errors.md b/src/current/v24.3/common-errors.md index 5b6961044b1..9c377c9a272 100644 --- a/src/current/v24.3/common-errors.md +++ b/src/current/v24.3/common-errors.md @@ -106,7 +106,7 @@ When running a single-node CockroachDB cluster, an error about replicas failing E160407 09:53:50.337328 storage/queue.go:511 [replicate] 7 replicas failing with "0 of 1 store with an attribute matching []; likely not enough nodes in cluster" ~~~ -This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) to update your default zone configuration to expect only one node: +This happens because CockroachDB expects three nodes by default. If you do not intend to add additional nodes, you can stop this error by using [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) to update your default [zone configuration]({% link {{ page.version.version }}/configure-replication-zones.md %}) to expect only one node: {% include_cached copy-clipboard.html %} ~~~ shell @@ -222,3 +222,4 @@ Try searching the rest of our docs for answers or using our other [support resou - [StackOverflow](http://stackoverflow.com/questions/tagged/cockroachdb) - [CockroachDB Support Portal](https://support.cockroachlabs.com) - [Transaction retry error reference]({% link {{ page.version.version }}/transaction-retry-error-reference.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/configure-replication-zones.md b/src/current/v24.3/configure-replication-zones.md index 1a61cf26308..bcb8cb15084 100644 --- a/src/current/v24.3/configure-replication-zones.md +++ b/src/current/v24.3/configure-replication-zones.md @@ -24,10 +24,16 @@ For each of these objects you can control: This page explains how replication zones work and how to use the `ALTER ... CONFIGURE ZONE` statement to manage them. `CONFIGURE ZONE` is a subcommand of the [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}#configure-zone), [`ALTER TABLE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone), [`ALTER INDEX`]({% link {{ page.version.version }}/alter-index.md %}#configure-zone), [`ALTER PARTITION`]({% link {{ page.version.version }}/alter-partition.md %}#create-a-replication-zone-for-a-partition), and [`ALTER RANGE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) statements. +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} + {{site.data.alerts.callout_info}} To configure replication zones, a user must be a member of the [`admin` role]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or have been granted [`CREATE`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) or [`ZONECONFIG`]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges) privileges. To configure [`system` objects](#for-system-data), the user must be a member of the `admin` role. {{site.data.alerts.end}} +{{site.data.alerts.callout_danger}} +{% include {{ page.version.version }}/zone-configs/avoid-manual-zone-configs.md %} +{{site.data.alerts.end}} + ## Overview Every [range]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) in the cluster is part of a replication zone. Each range's zone configuration is taken into account as ranges are rebalanced across the cluster to ensure that any constraints are honored. @@ -59,7 +65,6 @@ In addition, CockroachDB stores internal [**system data**]({% link {{ page.versi Level | Description ------|------------ -------|------------ Cluster | The `default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone. System Range | CockroachDB comes with pre-configured replication zones for important system ranges, such as the "meta" and "liveness" ranges. If necessary, you can add replication zones for the "timeseries" range and other system ranges as well. Editing replication zones for system ranges may override settings from `default`. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

CockroachDB also comes with pre-configured replication zones for the internal `system` database and the `system.jobs` table, which stores metadata about long-running jobs such as schema changes and backups. @@ -67,11 +72,250 @@ System Range | CockroachDB comes with pre-configured replication zones for impor When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: -1. If there's a replication zone for the row, CockroachDB uses it. -1. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone. -1. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone. -1. If there's no applicable table replication zone, CockroachDB uses the database replication zone. -1. If there's no applicable database replication zone, CockroachDB uses the `default` cluster-wide replication zone. +1. If there's a replication zone [for the row](#create-a-replication-zone-for-a-partition) (a.k.a. [partition]({% link {{ page.version.version }}/partitioning.md %})), CockroachDB uses it. +1. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the [secondary index replication zone](#create-a-replication-zone-for-a-secondary-index). +1. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the [table replication zone](#create-a-replication-zone-for-a-table). +1. If there's no applicable table replication zone, CockroachDB uses the [database replication zone](#create-a-replication-zone-for-a-database). +1. If there's no applicable database replication zone, CockroachDB uses [the `default` cluster-wide replication zone](#view-the-default-replication-zone). + +As this example shows, CockroachDB chooses which replication zone applies to a specific schema object using a hierarchy of inheritance. For most users [who never modify their zone configurations](#why-manual-zone-config-management-is-not-recommended), this level of understanding is sufficient. + +However, there are nuances in how inheritance works that are not explained by this example. For more details about how zone config inheritance works, see the section [How zone config inheritance works](#how-zone-config-inheritance-works). + +### How zone config inheritance works + +As [discussed previously](#level-priorities), CockroachDB always uses the most granular replication zone available for each schema object (database, table, etc.). More-specific settings applied to schema objects at lower levels of the inheritance tree will always override the settings of objects above them in the tree. All configurations will therefore be modified versions of the [`default` range]({% link {{ page.version.version }}/configure-replication-zones.md %}#view-the-default-replication-zone), which acts as the root of the tree. This means that in practice, **any changes to a specific schema object's zone configurations are by definition user-initiated**, either via [Multi-region SQL abstractions]({% link {{ page.version.version }}/multiregion-overview.md %}) or [manual changes](#why-manual-zone-config-management-is-not-recommended). + +Because each zone config inherits all of its initial values from its parent object, **it only stores the values that differ from its parent**. Any fields that are unset will be looked up in the parent object’s zone configuration. This continues recursively up the inheritance hierarchy all the way to the `default` zone config. In practice, most values are cached for performance. + +For more information, see the following subsections: + +- [The zone config inheritance hierarchy](#the-zone-config-inheritance-hierarchy) +- [Zone config inheritance - example SQL session](#zone-config-inheritance-example-sql-session) +- [Why manual zone config management is not recommended](#why-manual-zone-config-management-is-not-recommended) + +#### The zone config inheritance hierarchy + +The hierarchy of inheritance for zone configs can be visualized using the following outline-style diagram, in which each level of indentation denotes an inheritance relationship. + +The only exception to this simple inheritance relationship is that due to a known limitation, sub-partitions do not inherit their values from their parent partitions. Instead, sub-partitions inherit their values from the parent table. For more information, see [cockroachdb/cockroach#75862](https://github.com/cockroachdb/cockroach/issues/75862). + +``` +- default + - database + - table + - index + - partition (row) 1 + - sub-partition 1.1 (NB. Sub-partitions inherit from tables, *not* partitions - see cockroachdb/cockroach#75862) + - sub-partition 1.1.1 + - sub-partition 1.2 + - table + - index + - index + - ... +``` + +The way the zone config inheritance hierarchy works can be thought of from a ["bottom-up"](#the-bottom-up-view-of-inheritance) or ["top-down"](#the-top-down-view-of-inheritance) perspective; both are useful, but which one is easier to understand may depend on what you are trying to do. + +The "bottom-up" view of how zone config inheritance works is useful when you [are troubleshooting unexpected replication behavior]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}). + +In this view, you start with the most granular schema object whose replication behavior currently interests you. In a troubleshooting session, this is the one you think is the source of the unexpected behavior. You start at the target schema object, looking at the specific state of its configuration to see what is different about it from its parent object (if anything), and work upwards from there. + +The "top-down" perspective of how zone config inheritance works can be useful when thinking about how the system works as a whole. + +From the whole-system perspective, the hierarchy of schema object zone configs can be thought of as a tree. From this point of view, the system does a depth-first traversal down the tree of schema objects. For each object, **it uses the value of the most specific modified field from the current object's zone config, unless or until it finds a modified value of that field on an object deeper in the tree**. For each field on the current schema object's zone config where it doesn't find any modified value, it uses the inherited value from the node's parent, which inherits from its parent, and so on all the way back up to the root of the tree ([the `default` range](#view-the-default-replication-zone)). + +The following diagram presents the same set of schema objects as the previous outline-style diagram, but using boxes and lines joined with arrows that represent the "top-down" view. + +Each box represents a schema object in the zone configuration inheritance hierarchy. Each solid line ends in an arrow that points from a parent object to its child object, which will inherit the parent's values unless those values are changed at the child level. The dotted lines between partitions and sub-partitions represent the known limitation mentioned previously that sub-partitions do not inherit their values from their parent partitions. Instead, sub-partitions inherit their values from the parent table. For more information about this limitation, see [cockroachdb/cockroach#75862](https://github.com/cockroachdb/cockroach/issues/75862). + +zone config inheritance diagram + +#### Zone config inheritance - example SQL session + +You can verify [the inheritance behavior described previously](#how-zone-config-inheritance-works) in a SQL session. + +Start a [demo cluster]({% link {{ page.version.version }}/cockroach-demo.md %}). Create a sample database and table: + +{% include_cached copy-clipboard.html %} +~~~ sql +CREATE DATABASE IF NOT EXISTS test; +USE test; +CREATE TABLE IF NOT EXISTS kv (k INT, v INT); +~~~ + +Next, manually set a zone configuration field at the database level. In this example, use `num_replicas`: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE test CONFIGURE ZONE USING num_replicas = 1; +~~~ + +Check that the child table `test.kv` inherits the value of `num_replicas` from its parent database. You should see output like the following: + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FROM TABLE test.kv; +~~~ + +~~~ + target | raw_config_sql +----------------+------------------------------------------- + DATABASE test | ALTER DATABASE test CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 14400, + | num_replicas = 1, + | constraints = '[]', + | lease_preferences = '[]' +(1 row) +~~~ + +Then, set the `num_replicas` field on the table to a different value than its parent database: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER TABLE kv CONFIGURE ZONE USING num_replicas = 5; +~~~ + +This overrides the value of `num_replicas` set by the parent `test` database. From this point forward, until you discard the changed settings at the table level, the table will no longer inherit those changed values from its parent database. + +In other words, **the value of this field on the table's zone config has diverged from its parent database, and all state on this field must be managed manually going forward**. This divergence in state between parent and child objects, and the necessity to manage it manually once it diverges, is the reason [why manual zone config management is not recommended](#why-manual-zone-config-management-is-not-recommended). + +Next, check the zone config for the table `test.kv`, and verify that its value of `num_replicas` has diverged: + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FROM TABLE kv; +~~~ + +~~~ + target | raw_config_sql +-----------+-------------------------------------- + TABLE kv | ALTER TABLE kv CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 14400, + | num_replicas = 5, + | constraints = '[]', + | lease_preferences = '[]' +(1 row) +~~~ + +Next, change the value of `num_replicas` for the `test` database again. Once again, choose a different value than its child table `test.kv` (which has `num_replicas = 5`). + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE test CONFIGURE ZONE USING num_replicas = 7; +~~~ + + The reason for doing this is to confirm the previous claim that "the value of this field on the table's zone config has diverged from its parent database, and all state on this field must be managed manually going forward". + +The following [`SHOW ZONE CONFIGURATION`]({% link {{ page.version.version }}/show-zone-configurations.md %}) statement confirms that the value of the `num_replicas` field on the `kv` table is no longer inherited from its parent database `test`. + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FROM TABLE kv; +~~~ + +~~~ + target | raw_config_sql +-----------+-------------------------------------- + TABLE kv | ALTER TABLE kv CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 14400, + | num_replicas = 5, + | constraints = '[]', + | lease_preferences = '[]' +(1 row) +~~~ + +Note that even if you manually change the value of `num_replicas` in `kv` to match the value of its parent `test`, further changes to `test` **will still not be propagated downward to `test.kv`**. Confirm this by running the following statements: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER TABLE test.kv CONFIGURE ZONE USING num_replicas = 7; +~~~ + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE test CONFIGURE ZONE USING num_replicas = 9; +~~~ + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FROM TABLE kv; +~~~ + +~~~ + target | raw_config_sql +-----------+-------------------------------------- + TABLE kv | ALTER TABLE kv CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 14400, + | num_replicas = 7, + | constraints = '[]', + | lease_preferences = '[]' +(1 row) +~~~ + +One way to think about this from a programming perspective is that a ["dirty bit"](https://en.wikipedia.org/wiki/Dirty_bit) has been set on the `num_replicas` field in the `kv` table's zone config. + +However, you can confirm that other fields on `test.kv` which have not been modified from their default values still inherit from the parent database `test` as expected by changing the value of `gc.ttlseconds` on the `test` database. + +Change the value of `gc.ttlseconds` on the `test` database: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER DATABASE test CONFIGURE ZONE USING gc.ttlseconds = 600; +~~~ + +Next, confirm that the value of `gc.ttlseconds` from the `test` database is inherited by the `test.kv` table as expected: + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FROM TABLE test.kv; +~~~ + +~~~ + target | raw_config_sql +-----------------------+-------------------------------------------------- + TABLE test.public.kv | ALTER TABLE test.public.kv CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 600, + | num_replicas = 7, + | constraints = '[]', + | lease_preferences = '[]' +(1 row) +~~~ + +To return the `test.kv` table to a state where it goes back to inheriting all of its values from its parent database `test`, use the [`ALTER TABLE ... CONFIGURE ZONE DISCARD`]({% link {{ page.version.version }}/alter-table.md %}#remove-a-replication-zone) statement: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER TABLE test.kv CONFIGURE ZONE DISCARD; +~~~ + +As you can see from this example, these kinds of manual "tweaks" to zone configurations can leave the cluster in a state where its operators are managing its replication in a very manual and error-prone fashion. This example shows why, for most users, [manual zone config management is not recommended](#why-manual-zone-config-management-is-not-recommended). Instead, most users should use [Multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}). + +#### Why manual zone config management is not recommended + +Reasons to avoid modifying zone configurations manually using [`CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-database.md %}#configure-zone) include: + +- It is difficult to do proper auditing and change management of manually altered zone configurations. +- It is easy to introduce logic errors (a.k.a., bugs) and end up in a state where your replication is not behaving as it is "supposed to". +- Manual zone config modifications must be entirely maintained by the user with no help from the system. Such manual changes must be fully overwritten on each configuration change in order to take effect; this introduces another avenue for error. + +Given the [previous description of how zone config inheritance works](#how-zone-config-inheritance-works), if you [manually modify zone configs using `CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-database.md %}#configure-zone), you will see the following behavior: + +- If you set some field _F_ at the database level, the updated value of that field will propagate down the inheritance tree to all of the database's child tables. +- However, if you then edit that same field _F_ on a child table, the table will no longer inherit its value of _F_ from its parent database. Instead, **the field _F_ across these two schema objects is now in a diverged state**. This means that any changes to the value of _F_ at the database level **no longer propagate downward to the child table**, which has its own value of _F_. +- In other words, once you touch any schema object _O_ to manually edit some field _F_, a "dirty bit" is set on _O.F_, and CockroachDB will no longer modify _O.F_ without user intervention. **Instead, you are now responsible for managing the state of _O.F_ via direct calls to [`ALTER DATABASE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-database.md %}#configure-zone) anytime something needs to change in your configuration**. + +Avoiding this error-prone manual state management was the motivation behind the development of the abstractions provided by [Multi-region SQL statements]({% link {{ page.version.version }}/multiregion-overview.md %}) and [Multi-region Zone Config Extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}). ## Manage replication zones @@ -136,7 +380,7 @@ See [Cluster Topology]({% link {{ page.version.version }}/recommended-production ### Troubleshooting zone constraint violations -To see if any of the data placement constraints defined in your replication zone configurations are being violated, use the `system.replication_constraint_stats` report as described in [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}). +{% include {{ page.version.version }}/see-zone-config-troubleshooting-guide.md %} ## View replication zones @@ -690,4 +934,5 @@ There's no need to make zone configuration changes; by default, the cluster is c - [`SHOW PARTITIONS`]({% link {{ page.version.version }}/show-partitions.md %}) - [SQL Statements]({% link {{ page.version.version }}/sql-statements.md %}) - [Table Partitioning]({% link {{ page.version.version }}/partitioning.md %}) -- [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) +- [Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint) +- [Troubleshoot Replication Zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/demo-low-latency-multi-region-deployment.md b/src/current/v24.3/demo-low-latency-multi-region-deployment.md index 97102a0b8ef..2025a5d8c49 100644 --- a/src/current/v24.3/demo-low-latency-multi-region-deployment.md +++ b/src/current/v24.3/demo-low-latency-multi-region-deployment.md @@ -68,7 +68,7 @@ To determine which nodes are in which regions, you will need to refer to two (2) Here is the output of `\demo ls` from the SQL shell. {% include_cached copy-clipboard.html %} -~~~ sql +~~~ > \demo ls ~~~ @@ -145,7 +145,7 @@ Follow these steps to start 3 instances of MovR. Each instance is pointed at a n {% include_cached copy-clipboard.html %} ~~~ sql - CREATE DATABASE movr; + CREATE DATABASE IF NOT EXISTS movr; ~~~ 1. Open a second terminal and run the command below to populate the MovR data set. The options are mostly self-explanatory. We limit the application to 1 thread because using multiple threads quickly overloads this small demo cluster's ability to ingest data. As a result, loading the data takes about 90 seconds on a fast laptop. diff --git a/src/current/v24.3/local-testing.md b/src/current/v24.3/local-testing.md index 3ab1300adb9..871c4d77d11 100644 --- a/src/current/v24.3/local-testing.md +++ b/src/current/v24.3/local-testing.md @@ -31,8 +31,8 @@ We recommend the following additional [cluster settings]({% link {{ page.version | `jobs.registry.interval.cancel` | `180s` | Timing of an internal task that queries the [jobs]({% link {{ page.version.version }}/show-jobs.md %}) table. For testing, the default is too fast. | | `jobs.retention_time` | `15s` | More [schema changes]({% link {{ page.version.version }}/online-schema-changes.md %}) create more [jobs]({% link {{ page.version.version }}/show-jobs.md %}), which affects job query performance. We don’t need to retain jobs during testing and can set a more aggressive delete policy. | | `sql.stats.automatic_collection.enabled` | `false` | Turn off [statistics]({% link {{ page.version.version }}/show-statistics.md %}) collection, since automatic statistics contribute to table contention alongside schema changes. Each schema change triggers an asynchronous auto statistics job. | -| `ALTER RANGE default CONFIGURE ZONE USING "gc.ttlseconds"` | `600` | Faster descriptor cleanup. For more information, see [`ALTER RANGE`]({% link {{ page.version.version }}/alter-range.md %}). | -| `ALTER DATABASE system CONFIGURE ZONE USING "gc.ttlseconds"` | `600` | Faster jobs table cleanup. For more information, see [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}). | +| `ALTER RANGE default CONFIGURE ZONE USING "gc.ttlseconds"` | `600` | Faster descriptor cleanup. For more information, see [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone). | +| `ALTER DATABASE system CONFIGURE ZONE USING "gc.ttlseconds"` | `600` | Faster jobs table cleanup. For more information, see [`ALTER DATABASE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-database.md %}#configure-zone). | To change all of the settings described above at once, run the following SQL statements: diff --git a/src/current/v24.3/migrate-to-multiregion-sql.md b/src/current/v24.3/migrate-to-multiregion-sql.md index bd5838a1480..18c91005157 100644 --- a/src/current/v24.3/migrate-to-multiregion-sql.md +++ b/src/current/v24.3/migrate-to-multiregion-sql.md @@ -307,3 +307,4 @@ SHOW ZONE CONFIGURATION FROM TABLE promo_codes; - [Low Latency Reads and Writes in a Multi-Region Cluster]({% link {{ page.version.version }}/demo-low-latency-multi-region-deployment.md %}) - [Configure Replication Zones]({% link {{ page.version.version }}/configure-replication-zones.md %}) - [Non-voting replicas]({% link {{ page.version.version }}/architecture/replication-layer.md %}#non-voting-replicas) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/monitoring-and-alerting.md b/src/current/v24.3/monitoring-and-alerting.md index a7880f7c02a..92100759c6b 100644 --- a/src/current/v24.3/monitoring-and-alerting.md +++ b/src/current/v24.3/monitoring-and-alerting.md @@ -195,7 +195,9 @@ The critical nodes status endpoint is used to: - Check if any of your nodes are in a critical state. A node is _critical_ if that node becoming unreachable would cause [replicas to become unavailable]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#replication-status). - Check if any ranges are [under-replicated or unavailable]({% link {{ page.version.version }}/ui-cluster-overview-page.md %}#replication-status). This is useful when determining whether a node is ready for [decommissioning]({% link {{ page.version.version }}/node-shutdown.md %}#decommissioning). -- Check if any of your cluster's data placement constraints (set via [multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}) or direct [configuration of replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %})) are being violated. This is useful when implementing [data domiciling]({% link {{ page.version.version }}/data-domiciling.md %}). +- Check if any of your cluster's data placement constraints (set via [multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}) or direct [configuration of replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %})) are being violated. This is useful when implementing [data domiciling]({% link {{ page.version.version }}/data-domiciling.md %}) or [troubleshooting zone configurations]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) generally. + +If you find under-replicated ranges or constraint violations, you will need to [Troubleshoot your replication zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}). {{site.data.alerts.callout_info}} This HTTP status endpoint supersedes the deprecated [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) SQL API. Due to architectural changes in CockroachDB, the SQL queries described on that page will not result in correct output. @@ -358,7 +360,9 @@ In other words, this tells the ranges that "where you are now is *not* where you The critical nodes endpoint should now report a constraint violation in the `violatingConstraints` field of the response, similar to the one shown below. {{site.data.alerts.callout_success}} -You can also use the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to find out more information about the ranges that are in violation of constraints. +Use the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to find out more information about the ranges that are in violation of constraints. + +In a real life constraint violation scenario, you will need to [Troubleshoot your replication zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}). {{site.data.alerts.end}} {% include_cached copy-clipboard.html %} @@ -534,7 +538,9 @@ Once the statement above executes, the cluster will rebalance so that it's stori The critical nodes endpoint should now report ranges in the `underReplicated` field of the response, similar to the one shown below. {{site.data.alerts.callout_success}} -You can also use the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to find out more information about the under-replicated ranges. +Use the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to find out more information about the under-replicated ranges. + +In a real life under-replication scenario, you may need to [Troubleshoot your replication zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}). {{site.data.alerts.end}} {% include_cached copy-clipboard.html %} @@ -770,6 +776,13 @@ ALTER DATABASE movr ALTER LOCALITY REGIONAL IN "us-east1" CONFIGURE ZONE USING n The critical nodes endpoint should now report that all of the cluster's nodes are critical by listing them in the `criticalNodes` field of the response. +{{site.data.alerts.callout_success}} +Use the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to find out more information about the ranges in critical localities. + +In a real life critical localities scenario, you may need to [Troubleshoot your replication zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}). +{{site.data.alerts.end}} + + {% include_cached copy-clipboard.html %} ~~~ shell curl -X POST http://localhost:8080/_status/critical_nodes @@ -1216,3 +1229,4 @@ Currently, not all events listed have corresponding alert rule definitions avail - [Local Deployment]({% link {{ page.version.version }}/start-a-local-cluster.md %}) - [Third-Party Monitoring Integrations]({% link {{ page.version.version }}/third-party-monitoring-tools.md %}) - [Metrics]({% link {{ page.version.version }}/metrics.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/multiregion-overview.md b/src/current/v24.3/multiregion-overview.md index 297c76404dd..d57df9a4a95 100644 --- a/src/current/v24.3/multiregion-overview.md +++ b/src/current/v24.3/multiregion-overview.md @@ -180,3 +180,4 @@ For more information, see [Zone Config Extensions]({% link {{ page.version.versi - [`SET SECONDARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#set-secondary-region) - [`ALTER DATABASE ... DROP SECONDARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#drop-secondary-region) - [Zone Config Extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/node-shutdown.md b/src/current/v24.3/node-shutdown.md index 4c549df1edc..59556b95075 100644 --- a/src/current/v24.3/node-shutdown.md +++ b/src/current/v24.3/node-shutdown.md @@ -301,7 +301,7 @@ To determine an appropriate termination grace period: ### Size and replication factor -Before decommissioning a node, make sure other nodes are available to take over the range replicas from the node. If fewer nodes are available than the replication factor, CockroachDB will automatically reduce the replication factor (for example, from 5 to 3) to try to allow the decommission to succeed. However, the replication factor will not be reduced lower than 3. If three nodes are not available, the decommissioning process will hang indefinitely until nodes are added or you update the zone configurations to use a replication factor of 1. +Before decommissioning a node, make sure other nodes are available to take over the range replicas from the node. If fewer nodes are available than the replication factor, CockroachDB will automatically reduce the replication factor (for example, from 5 to 3) to try to allow the decommission to succeed. However, the replication factor will not be reduced lower than 3. If three nodes are not available, the decommissioning process will hang indefinitely until nodes are added or you update the [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_replicas) to use a replication factor of 1. Note that when you decommission a node and immediately add another node, CockroachDB does **not** simply move all of the replicas from the decommissioned node to the newly added node. Instead, replicas are placed across all nodes in the cluster. This speeds up the decommissioning process by spreading the load. The new node will eventually "catch up" with the rest of the cluster. diff --git a/src/current/v24.3/partitioning.md b/src/current/v24.3/partitioning.md index a313c97134a..6e0acd4b000 100644 --- a/src/current/v24.3/partitioning.md +++ b/src/current/v24.3/partitioning.md @@ -15,8 +15,8 @@ CockroachDB allows you to define table partitions, giving you row-level control Table partitioning helps you reduce latency and cost: -- **Geo-partitioning** allows you to keep user data close to the user, which reduces the distance that the data needs to travel, thereby **reducing latency**. To geo-partition a table, define location-based partitions while creating a table, create location-specific zone configurations, and apply the zone configurations to the corresponding partitions. -- **Archival-partitioning** allows you to store infrequently-accessed data on slower and cheaper storage, thereby **reducing costs**. To archival-partition a table, define frequency-based partitions while creating a table, create frequency-specific zone configurations with appropriate storage devices constraints, and apply the zone configurations to the corresponding partitions. +- **Geo-partitioning** allows you to keep user data close to the user, which reduces the distance that the data needs to travel, thereby **reducing latency**. To geo-partition a table, define location-based partitions while creating a table, create location-specific [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}), and apply the zone configurations to the corresponding partitions. +- **Archival-partitioning** allows you to store infrequently-accessed data on slower and cheaper storage, thereby **reducing costs**. To archival-partition a table, define frequency-based partitions while creating a table, create frequency-specific [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) with appropriate storage devices constraints, and apply the zone configurations to the corresponding partitions. ## How it works @@ -109,6 +109,7 @@ On their own, partitions are inert and simply apply a label to the rows of the t CockroachDB uses the most granular zone config available. Zone configs that target a partition are considered more granular than those that target a table or index, which in turn are considered more granular than those that target a database. +For more information about how zone config inheritance works, see [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}#level-priorities). {% include {{page.version.version}}/sql/querying-partitions.md %} @@ -688,7 +689,7 @@ Other databases use partitioning for three additional use cases: secondary index - **Changes to secondary indexes:** CockroachDB solves these changes through online schema changes. Online schema changes are a superior feature to partitioning because they require zero-downtime and eliminate the potential for consistency problems. - **Sharding:** CockroachDB automatically shards data as a part of its distributed database architecture. - **Bulk Loading & Deleting:** CockroachDB does not have a feature that supports this use case as of now. -- **Logical structure of partitions:** CockroachDB uses the partitioning concept to allow users to logically subdivide a single physical table to enable independent configuration of those partitions using zone configurations. Other databases sometimes implement partitioning as the logical union of physically separate tables. This difference means that CockroachDB is able to permit inexpensive repartitioning in contrast to other databases. Consequently, CockroachDB is unable to provide the same bulk data deletion operations over table partitions that other databases achieve by physically dropping the underlying table represented by the partition. +- **Logical structure of partitions:** CockroachDB uses the partitioning concept to allow users to logically subdivide a single physical table to enable independent configuration of those partitions using [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). Other databases sometimes implement partitioning as the logical union of physically separate tables. This difference means that CockroachDB is able to permit inexpensive repartitioning in contrast to other databases. Consequently, CockroachDB is unable to provide the same bulk data deletion operations over table partitions that other databases achieve by physically dropping the underlying table represented by the partition. ## Known limitations @@ -701,3 +702,4 @@ Other databases use partitioning for three additional use cases: secondary index - [`CREATE TABLE`]({% link {{ page.version.version }}/create-table.md %}) - [`ALTER TABLE`]({% link {{ page.version.version }}/alter-table.md %}) - [Computed Columns]({% link {{ page.version.version }}/computed-columns.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/query-replication-reports.md b/src/current/v24.3/query-replication-reports.md index 475624517b8..fc36f796a5d 100644 --- a/src/current/v24.3/query-replication-reports.md +++ b/src/current/v24.3/query-replication-reports.md @@ -522,3 +522,4 @@ To give another example, let's say your cluster were similar to the one shown ab - [`PARTITION BY`]({% link {{ page.version.version }}/partitioning.md %}) - [`CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone) - [Start a node]({% link {{ page.version.version }}/cockroach-start.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version }}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/restore.md b/src/current/v24.3/restore.md index e8aac279600..10f26aaf49d 100644 --- a/src/current/v24.3/restore.md +++ b/src/current/v24.3/restore.md @@ -25,9 +25,9 @@ For details on restoring across versions of CockroachDB, see [Restoring Backups - `RESTORE` only supports backups taken on a cluster on a specific major version into a cluster that is on the same version or the next major version. Refer to the [Restoring Backups Across Versions]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}) page for more detail. - `RESTORE` is a blocking statement. To run a restore job asynchronously, use the [`DETACHED`](#detached) option. - `RESTORE` no longer requires an {{ site.data.products.enterprise }} license, regardless of the options passed to it or to the backup it is restoring. -- [Zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) present on the destination cluster prior to a restore will be **overwritten** during a [cluster restore]({% link {{ page.version.version }}/restore.md %}#full-cluster) with the zone configurations from the [backed up cluster]({% link {{ page.version.version }}/backup.md %}#back-up-a-cluster). If there were no customized zone configurations on the cluster when the backup was taken, then after the restore the destination cluster will use the zone configuration from the [`RANGE DEFAULT` configuration]({% link {{ page.version.version }}/configure-replication-zones.md %}#view-the-default-replication-zone). - You cannot restore a backup of a multi-region database into a single-region database. - When the [`exclude_data_from_backup`]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#exclude-a-tables-data-from-backups) parameter is set on a table, the table will not contain row data when restored. +- {% include {{ page.version.version }}/backups/zone-configs-overwritten-during-restore.md %} ## Required privileges diff --git a/src/current/v24.3/show-backup.md b/src/current/v24.3/show-backup.md index d7ecd78a5d9..fe1849f69e1 100644 --- a/src/current/v24.3/show-backup.md +++ b/src/current/v24.3/show-backup.md @@ -77,7 +77,7 @@ Field | Value | Description `rows` | [`INT`]({% link {{ page.version.version }}/int.md %}) | Number of rows in tables that are part of the backup. `create_statement` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The `CREATE` statement used to create [table(s)]({% link {{ page.version.version }}/create-table.md %}), [view(s)]({% link {{ page.version.version }}/create-view.md %}), or [sequence(s)]({% link {{ page.version.version }}/create-sequence.md %}) that are stored within the backup. This displays when `SHOW BACKUP SCHEMAS` is used. Note that tables with references to [foreign keys]({% link {{ page.version.version }}/foreign-key.md %}) will only display foreign key constraints if the table to which the constraint relates to is also included in the backup. `is_full_cluster` | [`BOOL`]({% link {{ page.version.version }}/bool.md %}) | Whether the backup is of a full cluster or not. -`regions` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}) statement(s) used to configure the database with [multi-region zone configurations]({% link {{ page.version.version }}/multiregion-overview.md %}#database-regions), if they exist. If the database is a part of a single region cluster configuration, `NULL` will show. +`regions` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The [`ALTER DATABASE`]({% link {{ page.version.version }}/alter-database.md %}) statement(s) used to configure [multiple database regions]({% link {{ page.version.version }}/multiregion-overview.md %}#database-regions), if they exist. If the database is a part of a single region cluster configuration, `NULL` will show. `file_bytes` | [`INT`]({% link {{ page.version.version }}/int.md %}) | (With the `check_files` option only) The estimated bytes in external storage for a particular table object. This is the physical bytes that a given table object is taking up. For example, when the files are written to disk in storage they could be compressed. If you total all file bytes, the result is the physical bytes in your storage location. Note that for smaller tables the byte size in `file_bytes` may be larger than `size_bytes` because of the overhead required to create an SST file. `path` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | The list of the [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups)'s subdirectories. This field is returned for `SHOW BACKUPS IN collectionURI` only. The path format is `//-`. diff --git a/src/current/v24.3/show-create.md b/src/current/v24.3/show-create.md index 8ae0c86f859..c3b340f0802 100644 --- a/src/current/v24.3/show-create.md +++ b/src/current/v24.3/show-create.md @@ -96,7 +96,7 @@ To return just the `create_statement` value: ~~~ {{site.data.alerts.callout_info}} -`SHOW CREATE TABLE` also lists any partitions and zone configurations defined on primary and secondary indexes of a table. If partitions are defined, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. +`SHOW CREATE TABLE` also lists any partitions and [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}) defined on primary and secondary indexes of a table. If partitions are defined, but no zones are configured, the `SHOW CREATE TABLE` output includes a warning. {{site.data.alerts.end}} ### Show the `CREATE TABLE` statement for a table with a hidden column diff --git a/src/current/v24.3/show-zone-configurations.md b/src/current/v24.3/show-zone-configurations.md index 1df6ab75243..b827c31bc36 100644 --- a/src/current/v24.3/show-zone-configurations.md +++ b/src/current/v24.3/show-zone-configurations.md @@ -51,7 +51,7 @@ You can also use [`SHOW CREATE TABLE`]({% link {{ page.version.version }}/show-c ### View the replication zone for an index -To control replication for a specific index, use the `ALTER INDEX ... CONFIGURE ZONE` statement to define the relevant values (other values will be inherited from the parent zone): +To control replication for a specific index, use the [`ALTER INDEX ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-index.md %}#configure-zone) statement to define the relevant values (other values will be inherited from the parent zone): {% include_cached copy-clipboard.html %} ~~~ sql @@ -94,3 +94,4 @@ CONFIGURE ZONE 1 - [`ALTER RANGE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-range.md %}#configure-zone) - [`ALTER TABLE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone) - [SQL Statements]({% link {{ page.version.version }}/sql-statements.md %}) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}) diff --git a/src/current/v24.3/take-and-restore-locality-aware-backups.md b/src/current/v24.3/take-and-restore-locality-aware-backups.md index bd921f2674b..6c4558383a3 100644 --- a/src/current/v24.3/take-and-restore-locality-aware-backups.md +++ b/src/current/v24.3/take-and-restore-locality-aware-backups.md @@ -92,7 +92,7 @@ Specifying both locality tier pairs (e.g., `region=us-east,az=az1`) from the out Given a list of URIs that together contain the locations of all of the files for a single [locality-aware backup](#create-a-locality-aware-backup), [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) can read in that backup. Note that the list of URIs passed to [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) may be different from the URIs originally passed to [`BACKUP`]({% link {{ page.version.version }}/backup.md %}). This is because it's possible to move the contents of one of the parts of a locality-aware backup (i.e., the files written to that destination) to a different location, or even to consolidate all the files for a locality-aware backup into a single location. -When restoring a [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups), the cluster data is restored first, then the system table data "as is." This means that the restored zone configurations can point to regions that do not have active nodes in the new cluster. For example, if your full backup has the following [zone configurations]({% link {{ page.version.version }}/alter-partition.md %}#create-a-replication-zone-for-a-partition): +When restoring a [full backup]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#full-backups), the cluster data is restored first, then the system table data "as is." This means that the restored [zone configurations]({% link {{ page.version.version }}/alter-partition.md %}#create-a-replication-zone-for-a-partition) can point to regions that do not have active nodes in the new cluster. For example, if your full backup has the following zone configurations: ~~~ sql ALTER PARTITION europe_west OF INDEX movr.public.rides@rides_pkey \ diff --git a/src/current/v24.3/troubleshoot-replication-zones.md b/src/current/v24.3/troubleshoot-replication-zones.md new file mode 100644 index 00000000000..5d3ad9efe97 --- /dev/null +++ b/src/current/v24.3/troubleshoot-replication-zones.md @@ -0,0 +1,231 @@ +--- +title: Troubleshoot Replication Zones +summary: Troubleshooting guide for replication zones, which control the number and location of replicas for specific sets of data. +keywords: ttl, time to live, availability zone +toc: true +docs_area: manage +--- + +This page has instructions showing how to troubleshoot scenarios where you believe replicas are not behaving as specified by your [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). + +{{site.data.alerts.callout_danger}} +{% include {{ page.version.version }}/zone-configs/avoid-manual-zone-configs.md %} +{{site.data.alerts.end}} + +## Prerequisites + +This page assumes you have read and understood the following: + +- [Replication controls > Replication zone levels]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-levels), which describes how the inheritance hierarchy of replication zones works. **This is critical to understand for troubleshooting.** +- [Monitoring and alerting > Critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint), which is used to monitor if any of your cluster's ranges are under-replicated, or if your data placement constraints are being violated. +- [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}), which is the SQL statement used to view details about the replication zone configuration of various schema objects. + +## Types of problems + +The most common types of problems you may encounter when [manually configuring replication zones]({% link {{ page.version.version }}/configure-replication-zones.md %}#manage-replication-zones) are: + +1. The _replica location_ problem, a.k.a., "The replicas are not _where_ they should be". For replica location problems, the following zone config variables may be involved: + - [`constraints`]({% link {{ page.version.version }}/configure-replication-zones.md %}#constraints) + - [`lease_preferences`]({% link {{ page.version.version }}/configure-replication-zones.md %}#lease_preferences) + - [`voter_constraints`]({% link {{ page.version.version }}/configure-replication-zones.md %}#voter_constraints) + - [`global_reads`]({% link {{ page.version.version }}/configure-replication-zones.md %}#global_reads) +1. The _replica state_ problem_, a.k.a., "The replicas are not _how_ they should be". For replica state problems, the following zone config variables may be involved: + - [`range_min_bytes`]({% link {{ page.version.version }}/configure-replication-zones.md %}#range-min-bytes) + - [`range_max_bytes`]({% link {{ page.version.version }}/configure-replication-zones.md %}#range-max-bytes) + - [`gc.ttlseconds`]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) + - [`num_replicas`]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_replicas) + - [`num_voters`]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_voters) + +The replica location problem is the most common reason for a troubleshooting session. It is most often caused by misconfiguration introduced by manually configuring replication zones, which is [why manual zone config management is not recommended]({% link {{ page.version.version }}/configure-replication-zones.md %}#why-manual-zone-config-management-is-not-recommended). Most users should use the [Multi-region SQL statements]({% link {{ page.version.version }}/multiregion-overview.md %}). If further customization is needed, see [Multi-region Zone Config Extensions]({% link {{ page.version.version }}/zone-config-extensions.md %}). + +{{site.data.alerts.callout_info}} +If you just did a [cluster restore]({% link {{ page.version.version }}/restore.md %}#full-cluster) and are seeing problems with your zone configs, it may be because [zone configs are overwritten by a cluster restore](#zone-configs-are-overwritten-during-a-cluster-restore). +{{site.data.alerts.end}} + +## Troubleshooting steps + +Follow the steps below to determine which schema objects (if any) have zone configurations that are misconfigured. + +The examples in the steps assume a local multi-region [`cockroach demo`]({% link {{ page.version.version }}/cockroach-demo.md %}) cluster started using the following command: + +{% include_cached copy-clipboard.html %} +~~~ shell +cockroach demo --global --nodes 9 --insecure +~~~ + +Next, execute the following statements to set the [database regions]({% link {{ page.version.version }}/multiregion-overview.md %}#database-regions) for the [`movr` database]({% link {{ page.version.version }}/movr.md %}#the-movr-database): + +{% include_cached copy-clipboard.html %} +~~~ sql + ALTER DATABASE movr SET PRIMARY REGION "us-east1"; + ALTER DATABASE movr ADD REGION "us-west1"; + ALTER DATABASE movr ADD REGION "europe-west1"; +~~~ + + + +### Step 1. Start with a target schema object + +Start by looking at the zone configuration for the schema object that you think may be misconfigured. Depending on the [type of problem](#types-of-problems), one way you may have come to this conclusion is by [monitoring the critical nodes endpoint](#monitoring). + +Use the [`SHOW ZONE CONFIGURATION`]({% link {{ page.version.version }}/show-zone-configurations.md %}) statement to inspect the target object's zone configurations. + +For example, this is the zone configuration for the `movr.rides` table: + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW ZONE CONFIGURATION FOR TABLE movr.users; +~~~ + +~~~ +----------------+------------------------------------------------------------------------------------------- + DATABASE movr | ALTER DATABASE movr CONFIGURE ZONE USING + | range_min_bytes = 134217728, + | range_max_bytes = 536870912, + | gc.ttlseconds = 14400, + | num_replicas = 5, + | num_voters = 3, + | constraints = '{+region=europe-west1: 1, +region=us-east1: 1, +region=us-west1: 1}', + | voter_constraints = '[+region=us-east1]', + | lease_preferences = '[[+region=us-east1]]' +(1 row) +~~~ + +In this case, the output is expected for a multi-region cluster spread across 9 nodes that was configured using [Multi-region SQL]({% link {{ page.version.version }}/multiregion-overview.md %}). Since nothing has been manually modified, the output shows that `movr.users` is using the zone configuration from its parent `movr` database. For more information about how this works, see [How zone config inheritance works](#how-zone-config-inheritance-works). + +However, if the zone configuration had been manually modified, there could be inconsistencies in the output that would show a misconfiguration. + +For example: + +- If the [type of problem](#types-of-problems) were a [constraint violation](#monitoring), you'd want to check that the values in [`constraints`]({% link {{ page.version.version }}/configure-replication-zones.md %}#constraints), [`voter_constraints`]({% link {{ page.version.version }}/configure-replication-zones.md %}#voter_constraints), and [`lease_preferences`]({% link {{ page.version.version }}/configure-replication-zones.md %}#lease_preferences) were not logically inconsistent, which would cause constraint violations. +- If the [type of problem](#types-of-problems) were an [under-replicated range](#monitoring), you'd want to check the values of [`num_replicas`]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_replicas) and [`num_voters`]({% link {{ page.version.version }}/configure-replication-zones.md %}#num_voters). Modifying these values can cause under-replication. +- If [range splits are failing]({% link {{ page.version.version }}/common-errors.md %}#split-failed-while-applying-backpressure-are-rows-updated-in-a-tight-loop), the value of [`gc.ttlseconds`]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) may be too high. + +If the zone configuration for the target schema object looks good, [move to the next step](#step2). + +If the zone configuration does not look right, repair it now using [`ALTER TABLE ... CONFIGURE ZONE`]({% link {{ page.version.version }}/alter-table.md %}#configure-zone). You can set the field to another value, but often the best thing to do is to discard the changed settings using [`ALTER TABLE ... CONFIGURE ZONE DISCARD`]({% link {{ page.version.version }}/alter-table.md %}#remove-a-replication-zone) so that it returns to inheriting values from its parent object: + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER TABLE movr.users CONFIGURE ZONE DISCARD; +~~~ + + + +### Step 2. Move upward in the inheritance hierarchy as needed + +If the previous target schema object looked good, look at its parent schema object at the next level up in [the inheritance hierarchy]({% link {{ page.version.version }}/configure-replication-zones.md%}#how-zone-config-inheritance-works). + +This is the new target schema object. Return to [the previous step](#step1) and follow the instructions there. + +Continue this process recursively until you either find the misconfigured zone configuration or make it all the way up to [the `default` replication zone]({% link {{ page.version.version }}/configure-replication-zones.md %}#view-the-default-replication-zone) and confirm that all of your schema objects have the expected zone configurations. + +## Considerations + + + +### Monitor for ranges that are under-replicated or violating constraints + +Monitor the output of the [critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint) to see if you have ranges that are under-replicated, or violating constraints. + +Using the range IDs from that endpoint, you can map from range IDs to schema objects as described below. This will give you [a target schema object to start from](#step-1). + +To monitor for under-replicated range IDs, see [Critical nodes endpoint > Replication status - under-replicated ranges]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#replication-status-under-replicated-ranges). + +To monitor for range IDs that are violating constraints, see [Critical nodes endpoint > Replication status - constraint violation]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#replication-status-constraint-violation). + +Once you have a range ID, you need to map from that ID to the name of a schema object that you can pass to [`SHOW ZONE CONFIGURATIONS`]({% link {{ page.version.version }}/show-zone-configurations.md %}). + +The following example query uses the [`SHOW RANGES`]({% link {{ page.version.version }}/show-ranges.md %}) statement to show, for each range ID, which tables and indexes use that range for their underlying storage. The query assumes the [`movr` schema]({% link {{ page.version.version }}/movr.md %}#schema) that is loaded by [`cockroach demo`]({% link {{ page.version.version }}/cockroach-demo.md %}), so you'll need to modify it to work with your schema. + +[XXX](XXX): GET HELP WITH THIS QUERY, OUTPUT SEEMS NOT QUITE RIGHT??? (or is it ok?) + +{% include_cached copy-clipboard.html %} +~~~ sql +WITH movr_tables AS (SHOW RANGES FROM DATABASE movr WITH TABLES), + movr_indexes AS (SHOW RANGES FROM DATABASE movr WITH INDEXES) + SELECT movr_indexes.range_id, + movr_tables.table_name, + movr_indexes.index_name + FROM movr_tables, movr_indexes + WHERE movr_tables.range_id = movr_indexes.range_id +ORDER BY movr_tables.range_id +~~~ + +In the following output, each range ID in the leftmost column maps to a table name and index name in the subsequent columns. For example, given this output, if the critical nodes endpoint said the constraint violation was for range ID 103, we would know to look at the zone configuration(s) for the `rides` table and the `rides_pkey` primary index. + +[XXX](XXX): EDIT THIS QUERY OUTPUT ONCE YOU GET A (HOPEFULLY) BETTER QUERY + +~~~ + range_id | table_name | index_name +-----------+----------------------------+------------------------------------------------ + 71 | promo_codes | promo_codes_pkey + 71 | user_promo_codes | promo_codes_pkey + 78 | vehicle_location_histories | vehicle_location_histories_pkey + 78 | promo_codes | vehicle_location_histories_pkey + 79 | user_promo_codes | user_promo_codes_pkey + 85 | vehicle_location_histories | rides_auto_index_fk_vehicle_city_ref_vehicles + 85 | rides | rides_auto_index_fk_vehicle_city_ref_vehicles + 87 | users | users_pkey + 88 | users | users_pkey + 89 | users | users_pkey + 90 | users | users_pkey + 91 | users | users_pkey + 92 | vehicles | vehicles_pkey + 92 | users | users_pkey + 92 | users | vehicles_pkey + 92 | vehicles | users_pkey + 93 | users | users_pkey + 94 | users | users_pkey + 95 | vehicles | vehicles_pkey + 96 | vehicles | vehicles_pkey + 97 | vehicles | vehicles_pkey + 98 | rides | vehicles_pkey + 98 | rides | vehicles_auto_index_fk_city_ref_users + 98 | vehicles | rides_pkey + 98 | rides | rides_pkey + 98 | vehicles | vehicles_pkey + 98 | vehicles | vehicles_auto_index_fk_city_ref_users + 99 | vehicles | vehicles_pkey + 100 | vehicles | vehicles_pkey + 101 | vehicles | vehicles_pkey + 102 | vehicles | vehicles_pkey + 103 | rides | rides_pkey + 104 | rides | rides_pkey + 105 | rides | rides_pkey + 106 | rides | rides_pkey + 106 | rides | rides_auto_index_fk_city_ref_users + 107 | rides | rides_pkey + 108 | rides | rides_pkey + 109 | rides | rides_pkey + 110 | rides | rides_pkey + 111 | users | users_pkey +(41 rows) +~~~ + +### Replication system priorities: data placement vs data durability + +As noted in [Data Domiciling with CockroachDB]({% link {{ page.version.version }}/data-domiciling.md %}): + +
+*[Zone configs]({% link {{ page.version.version }}/configure-replication-zones.md %}) can be used for data placement but these features were historically built for performance, not for domiciling. The [replication system]({% link {{ page.version.version }}/architecture/replication-layer.md %})'s top priority is to prevent the loss of data and it may override the zone configurations if necessary to ensure data durability.* +
+ +### Performance + +Changes you make to a schema object's zone configuration may take some time to be reflected in the schema object's actual state, and can result in an increase in CPU usage, IOPS, and network traffic while the cluster rebalances replicas to meet the provided constraints. This is especially true for larger clusters. + +For more information about how replica rebalancing works, see [Load-based replica rebalancing]({% link {{ page.version.version }}/architecture/replication-layer.md %}#load-based-replica-rebalancing). + +### Zone configs are overwritten during cluster restore + +{% include {{ page.version.version }}/backups/zone-configs-overwritten-during-restore.md %} + +For more information, see [`RESTORE` considerations]({% link {{ page.version.version }}/restore.md %}#considerations). + +## See also + +- [Troubleshooting overview]({% link {{ page.version.version }}/troubleshooting-overview.md %}) +- [Troubleshoot Self-Hosted Setup > Replication issues]({% link {{ page.version.version }}/cluster-setup-troubleshooting.md %}#replication-issues) +- [Replication Controls]({% link {{ page.version.version }}/configure-replication-zones.md %}) +- [Critical nodes status endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint): check the status of your cluster's data replication, data placement, and zone constraint conformance. diff --git a/src/current/v24.3/troubleshooting-overview.md b/src/current/v24.3/troubleshooting-overview.md index 9b21e2725f0..1ada8bdc479 100644 --- a/src/current/v24.3/troubleshooting-overview.md +++ b/src/current/v24.3/troubleshooting-overview.md @@ -20,7 +20,9 @@ If you experience an issue when using CockroachDB, try these steps to resolve th - If you see discrepancies in metrics, refer to [Differences in Metrics between Third-Party Monitoring Integrations and DB Console]({% link {{ page.version.version }}/differences-in-metrics-between-third-party-monitoring-integrations-and-db-console.md %}). -- [Replication Reports]({% link {{ page.version.version }}/query-replication-reports.md %}) let you query the status of your cluster's data replication, data placement, and zone constraint conformance. +- The [critical nodes endpoint]({% link {{ page.version.version }}/monitoring-and-alerting.md %}#critical-nodes-endpoint) lets you query the status of your cluster's data replication, data placement, and zone constraint conformance. + +- If you believe [replicas]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-replica) are not behaving as specified by your zone configurations, whether in terms of data placement or zone constraint conformance, see [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %}). - If you cannot resolve the issue yourself, the following tools can help you move forward: - [Support Resources]({% link {{ page.version.version }}/support-resources.md %}) identify ways you can get help with troubleshooting. diff --git a/src/current/v24.3/ui-debug-pages.md b/src/current/v24.3/ui-debug-pages.md index c75bcf60a0f..0758d18d462 100644 --- a/src/current/v24.3/ui-debug-pages.md +++ b/src/current/v24.3/ui-debug-pages.md @@ -32,7 +32,7 @@ Report | Description | Access level --------|-----|-------- [Custom Time Series Chart]({% link {{ page.version.version }}/ui-custom-chart-debug-page.md %}) | Create a custom chart of time series data. | All users. Problem Ranges | View ranges in your cluster that are unavailable, under-replicated, slow, paused, or have other problems. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). -Data Distribution and Zone Configs | View the distribution of table data across nodes and verify zone configuration. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). +Data Distribution and Zone Configs | View the distribution of table data across nodes and verify [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) or a SQL user with the [`VIEWCLUSTERMETADATA`]({% link {{ page.version.version }}/security-reference/authorization.md %}#viewclustermetadata) [system privilege]({% link {{ page.version.version }}/security-reference/authorization.md %}#supported-privileges). Statement Diagnostics History | Diagnostic bundles for all statements executed on the cluster. | On secure clusters, [`admin` user]({% link {{ page.version.version }}/security-reference/authorization.md %}#admin-role) only. ## Configuration diff --git a/src/current/v24.3/zone-config-extensions.md b/src/current/v24.3/zone-config-extensions.md index 980eae68074..7e01f0076a3 100644 --- a/src/current/v24.3/zone-config-extensions.md +++ b/src/current/v24.3/zone-config-extensions.md @@ -44,3 +44,4 @@ For instructions showing how to use Zone Config Extensions, see [Use Zone Config - [`SET LOCALITY`]({% link {{ page.version.version }}/alter-table.md %}#set-locality) - [`ADD REGION`]({% link {{ page.version.version }}/alter-database.md %}#add-region) - [`SET PRIMARY REGION`]({% link {{ page.version.version }}/alter-database.md %}#set-primary-region) +- [Troubleshoot Replication Zones]({% link {{ page.version.version}}/troubleshoot-replication-zones.md %})