Kiln » KilnSupportScripts Powershell Scripts to help monitor a Kiln environment. Contact Fog Creek support before using. Read More
Clone URL:  
Pushed to one repository · View In Graph Contained in tip

Merge

Changeset a141820a4935…

Parents 13315a17166d…

Parents 8087f38d2aea…

by Profile picture of User 476Ben McCormack <benm@fogcreek.com>

Changes to 4 files · Browse files at a141820a4935 Showing diff from parent 13315a17166d 8087f38d2aea Diff from another changeset...

Change 1 of 2 Show Entire File README.TXT Stacked
renamed from readMe.txt
 
 
 
 
 
1
2
3
 
10
11
12
13
14
15
 
 
 
16
17
18
19
20
 
21
22
 
 
 
23
24
 
25
26
 
 
 
 
 
 
27
28
29
30
 
 
 
31
32
 
 
 
 
 
33
 
 
 
 
 
 
34
 
 
 
 
 
 
 
 
35
36
37
38
39
40
 
41
42
 
 
43
44
45
46
47
48
49
50
 
51
52
53
54
55
 
 
56
57
 
 
 
 
58
59
 
 
 
 
 
60
61
 
 
62
63
 
 
 
 
 
 
 
 
 
64
65
 
 
 
 
 
 
 
 
 
 
 
66
67
 
 
 
68
69
 
70
71
72
73
 
 
 
 
 
 
1
2
3
4
5
6
7
 
14
15
16
 
 
 
17
18
19
20
21
22
23
 
24
25
 
26
27
28
29
 
30
31
 
32
33
34
35
36
37
38
39
40
 
41
42
43
44
 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
 
 
 
 
67
68
 
69
70
71
 
 
 
 
 
 
 
72
73
74
75
76
77
78
79
80
 
81
82
83
84
85
 
86
87
88
89
90
91
 
92
93
94
 
95
96
97
98
99
100
101
102
103
104
 
105
106
107
108
109
110
111
112
113
114
115
116
 
117
118
119
120
 
121
122
 
 
 
 
123
124
125
 
@@ -1,3 +1,7 @@
+ATTENTION: THESE SCRIPTS HAVE NOT BEEN TESTED IN A PRODUCTION ENVIRONMENT!!! +IF YOU FEEL YOU ARE HAVING ISSUES WITH YOUR KILN INSTALLATION, PLEASE CONTACT +FOG CREEK SUPPORT AT CUSTOMER-SERVICE@FOGCREEK.COM. +  This repository contains a series of Powershell scripts that can be used  to check up on Kiln, log environment data, restart the backend if necessary,  and even send email for alerts. @@ -10,64 +14,112 @@
    - checkBackend.ps1 Contains the Powershell functions to check Kiln   - test-checkBackend.ps1 Use this file to test functions - - run-checkBackend.example.ps1 This is an example script that uses the functions - in checkBackend.ps1. - - tools.ps1 Helpers for parsing the log file + - run-checkBackend.example.ps1 This is an example script that uses the + functions in checkBackend.ps1. + - tools.ps1 Helpers for parsing the log file      Here's a more detailed explanation for some of the files   -checkBackend.ps1 +# checkBackend.ps1   -This is the main file that contains the methods used by the other scripts. Nothing is actually executed when you simply run the file using . .\checkBackend.ps1 , but instead, the functions are loaded into memory. +This is the main file that contains the methods used by the other scripts. +Nothing is actually executed when you simply run the file using +`. .\checkBackend.ps1`, but instead, the functions are loaded into memory.   -test-checkBackend.ps1 +# test-checkBackend.ps1   -You can use this file to make sure your variables are set up correctly. You'll need to update the first line to correctly point to the checkBackend.ps1. Test-KilnVersion requires no parameters, so there's nothing we need to do here. However, Test-KilnDiff requires actual repository information, so we're likely going to want to go to the database to get this data. (You could use an existing /repo/{GUID}/diff/... URL that you've seen earlier, but this might be for a very large diff, and if possible, we'd like to use a much smaller diff). +You can use this file to make sure your variables are set up correctly. You'll +need to update the first line to correctly point to the checkBackend.ps1. +Test-KilnVersion requires no parameters, so there's nothing we need to do here. +However, Test-KilnDiff, which gets a diff from the Kiln Storage Service, +requires actual repository information, so we're likely going to want to go to +the database to get this data.    $sRepoGUID, $sChangeset1, and $sChangeset2   -What we're looking for is a relatively small diff (a few lines) in a repository that we don't expect to be deleted. This will help us ensure that our Test-KilnDiff method runs reliably without consuming too many resources. +What we're looking for is a relatively small diff (a few lines) in a repository +that we don't expect to be deleted. This will help us ensure that our +Test-KilnDiff method runs reliably without consuming too many resources.   -You can use the Kiln UI to find a changeset with a diff that appears relatively small. Once you find a changeset, view it in the Kiln UI, and not the partial changesetID in the URL, e.g. .../History/f181ad063566. Once you've noted the changesetID, run the following query in the Kiln database, updating the Where clause accordingly: +You can use the Kiln UI to find a changeset with a diff that appears relatively +small. Once you find a changeset, view it in the Kiln UI, and note the partial +changesetID in the URL, e.g. .../History/f181ad063566. Once you've noted the +changesetID, run the following query in the Kiln database, updating the Where +clause accordingly:   + Select Repo.sName, Repo.UUid, Changeset.sID, Changeset.sParentID1 + From Changeset + Inner Join ChangesetPush + On Changeset.ixChangeset = ChangesetPush.ixChangeset + Inner Join Repo On ChangesetPush.ixRepo = Repo.ixRepo + Where Left(sID,12) = 'f181ad063566'   +This is going to give you the UUid, sID, and sParentID1, which you'll map to +$sRepoGUID, $sChangeset1, and $sChangeset2, accordingly. + +Testing values in test-checkBackend.ps1: + +You can go ahead and update $logFile as well. It's not used in the test, but +if you copy variables over to run-checkBackend.example.ps1, they might as well +be the same.   -Select Repo.sName, Repo.UUid, Changeset.sID, Changeset.sParentID1 -From Changeset -Inner Join ChangesetPush On Changeset.ixChangeset = ChangesetPush.ixChangeset -Inner Join Repo On ChangesetPush.ixRepo = Repo.ixRepo -Where Left(sID,12) = 'f181ad063566' +$kilnRepositories is used to find miniredis.db, so update it accordingly.   -This is going to give you the UUid, sID, and sParentID1, which you'll map to $sRepoGUID, $sChangeset1, and $sChangeset2, accordingly. +Once all the values have been updated, navigate to the directory from the +powershell command line and run:   -Testing values - -You can go ahead and update $logFile as well. It's not used, in the test, but if you copy variables over to run-checkBackend.ps1, they might as well be the same. I don't believe the file needs to exist yet, but the directory probably needs to exist. - -Once all the values have been updated, navigate to the directory from the powershell command line and run: - -. .\test-checkBackend.ps1 + . .\test-checkBackend.ps1    You should see:    True  True +The size of miniredis.db is: 123 +123   -The first True indicates that it was able to ping the backend for version. The second True indicates that it was successfully able to run the Diff. If you get a false back for both, then either the backend is completely down or something is wrong with our script. +The first True indicates that it was able to ping the backend for version. +The second True indicates that it was successfully able to run the Diff. If +you get a false back for both, then either the backend is completely down or +something is wrong with our script.   -If you get a False back for the second value, it's likely an issue with the repo and changeset variables that you have. +If you get a False back for the second value, it's likely an issue with the +repo and changeset variables that you have. + +If you don't see a size for miniredis.db, check your $kilnRepositories +variable.   -Once you get two Trues, move to the next step. +Once you get a satisfactory output with two Trues and a size for miniredis.db, +move to the next step.   -run-checkBackend.ps1 +# run-checkBackend.example.ps1 + +This script is an example of what functionality is available in +checkBackend.ps1. You may not need each bit of functionality in your +environment, so remove sections of code as appropriate. + +I explain the variables for $sRepoGUID, $sChangeset1, and $sChangeset2 in the +above section for test-checkBackend.ps1, so read the relevant info if you +haven't already.   -This file is going to use the same variables as test-checkBackend.ps1, so go ahead and copy over the first 5 lines from test- and replace the first 5 lines in run-. +This file is going to use the same variables as test-checkBackend.ps1, so go +ahead and copy over the variable lines to run-checkBackend.example.ps1. + +The script begins with logging memory usage for Kiln processes and instead of +simply testing the services, this will actively log and will indeed restart +the storage service in Check-KilnBackend if either of the Test- methods fail. +That's why it's important to make sure your variables work in +test-checkBackend.ps1 before running this script. + +If you want, you can configure it to send you an email if a certain condition +is met, such as if a check of Kiln fails or if miniredis.db is too large.   -Instead of simply testing the services, this will actively log and will indeed restart the storage service in Check-KilnBackend if either of the Test- methods fail. That's why it's important to make sure your variables work in Test before running this script. +Once you've tested your variables in test- and have copied them over to run-, +you can test this script by navigating to the directory in powershell and +running:   -Once you've tested your variables in test- and have copied them over to run-, you can test this script by navigating to the directory in powershell and running: + . .\run-checkBackend.ps1   -. .\run-checkBackend.ps1 - -Once that executes, you should see that your log file has updated. If so, proceed to the next step. \ No newline at end of file
+Once that executes, you should see that your log file has updated. If it worked +successfully, you may want to have the script run every 15 minutes as a +scheduled Windows task. \ No newline at end of file
Change 1 of 1 Show Changes Only checkBackend.ps1 Stacked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
 
54
55
56
57
58
59
 
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
 $g_backendUrl = "http://localhost:56783/"  $g_kilnRepositories = "C:\KilnRepositories"    function Get-URL([string] $url){   (New-Object net.webclient).DownloadString($url)  }    function LogOrWrite($msg, $logFile = ""){   if ($logFile -ne "") {   write "$(Get-Date -format o) $msg" | out-file -append $logFile   }   else {   write $msg   }  }    function Test-KilnVersion(){   try {   $resp = Get-Url(($g_backendUrl + "version"))   return ($resp.length -gt 0)   }   catch {   return $False   }  }    function Test-KilnDiff([string] $sRepoGUID, [string] $sChangeset1, [string] $sChangeset2){   try {   $url =$g_backendUrl + "repo/" + $sRepoGUID + "/diff/" + $sChangeset1 + ":" + $sChangeset2 + "?format=json&ignorews=True&maxsize=100000&timeout=10"   $resp = Get-Url($url)   return ($resp.length -gt 0)   }   catch {   return $False   }  }    function Get-ProcessStats([string] $service, [string] $logFile){   try {   $measure = get-process $service -ErrorAction Stop | Measure-Object WorkingSet -sum   LogOrWrite "The process $service has $($measure.Count) instances with a total WorkingSet of: $($measure.Sum)" $logFile   }   catch {   LogOrWrite "The process $service is not currently running." $logFile   }  }    function Get-MiniRedisSize([string] $kilnRepositoriesDir = $g_kilnrepositories, [string] $logFile = ""){   try {   $kilnRepo = (Resolve-Path $kilnRepositoriesDir -ErrorAction Stop).path   }   catch {   LogOrWrite "Could not find Kiln Repositories Directory: $kilnRepositoriesDir" $logFile + return   }   try {   $miniRedis = (Resolve-Path ($kilnRepo + "\miniredis.db") -ErrorAction Stop).path   }   catch {   LogOrWrite "Could not find miniredis.db file in $kilnRepo" $logFile + return   }   $lenMiniRedis = (get-childitem -path $miniRedis).length   LogOrWrite "The size of miniredis.db is: $lenMiniRedis" $logFile   return $lenMiniRedis  }    function Check-KilnBackend([string] $sRepoGUID, [string] $sChangeset1, [string] $sChangeset2, [string] $logFile = ""){   if (-Not ((Test-KilnVersion) -and (Test-KilnDiff $sRepoGUID $sChangeset1 $sChangeset2))){   #Restart the Kiln storage Service because something bad happened   LogOrWrite "Something is wrong. Restarting Kiln Storage Service" $logFile   Restart-Service "KilnStorageService"   return $False   }   else {   LogOrWrite "Kiln backend is OK" $logFile   return $True   }  }
Change 1 of 1 Show Entire File run-checkBackend.example.ps1 Stacked
 
4
5
6
 
7
8
9
10
11
12
13
 
4
5
6
7
8
9
10
 
11
12
13
@@ -4,10 +4,10 @@
 $sChangeset2 = 'ed5f14291c3ab9d9bf24dee8e1b4761799509d15'  $logFile = "C:\code\checkKiln\checkBackend.log"  $kilnRepositories = "C:\KilnRepositories" +$maxMiniRedisSize = 3000000 #3MB  $smtpServer = "testmail.example.com"  $from = "ben@example.com"  $to = "ben@example.com" -$maxMiniRedisSize = 3000000 #3MB    Get-ProcessStats "backend" $logFile  Get-ProcessStats "memcached" $logFile
Change 1 of 1 Show Entire File test-checkBackend.ps1 Stacked
 
3
4
5
 
 
6
7
8
 
 
 
 
 
3
4
5
6
7
8
9
 
 
10
11
 
@@ -3,6 +3,9 @@
 $sChangeset1 = 'f181ad06356697b8676868fec36fbe13167081dd'  $sChangeset2 = 'ed5f14291c3ab9d9bf24dee8e1b4761799509d15'  $logFile = "C:\code\checkKiln\checkBackend.log" +$kilnRepositories = "C:\KilnRepositories" +$maxMiniRedisSize = 3000000 #3MB    Test-KilnVersion -Test-KilnDiff $sRepoGUID $sChangeset1 $sChangeset2 \ No newline at end of file
+Test-KilnDiff $sRepoGUID $sChangeset1 $sChangeset2 +Get-MiniRedisSize $kilnRepositories \ No newline at end of file