MonthJuly 2014

Octopus 2 – 1.6 to 2.x migration

Octopus 2 1.6 to 2.x migration

NOTE: I do not recomend using this method unless you have backups of everything related to your octopus installation, which in my case was 2 FOLDERS as well as an octopus made backup.
Use this tinformation at your own risk and I take no responsability if it does not work for you. (I was told I needed to add this incase someone broke something and pointed a finger)

Have your Octopus 2.x license key on hand.

Ever run into this when trying to upgrade from 1.6 to 2.x.

-------------------------------------------------------------------------------
Error: An error occurred when trying to import a batch of items of type RetentionPolicy: PUT vetoed by Raven.Bundles.UniqueConstraints.UniqueConstraintsPutTrigger because: Ensure unique constraint violated for fields: Name
-------------------------------------------------------------------------------
Full error details are available in the log files.
See: http://g.octopushq.com/LogFiles
Error: The previous command returned a non-zero exit code of: 100
Error: The command that failed was: "E:\Octopus\Octopus.Server.exe" migrate --instance "OctopusServer" --no-prompt --backup "E:\Installs\Octopus v1 backup\20140730-120334.octobak" --tentacle-port "10933"

Having a look in the OctopusServer.txt log file i see the following.

2014-07-30 15:58:43.5543 FATAL System.Exception: An error occurred when trying to import a batch of items of type RetentionPolicy: PUT vetoed by Raven.Bundles.UniqueConstraints.UniqueConstraintsPutTrigger because: Ensure unique constraint violated for fields: Name ---> Raven.Abstractions.Exceptions.OperationVetoedException: PUT vetoed by Raven.Bundles.UniqueConstraints.UniqueConstraintsPutTrigger because: Ensure unique constraint violated for fields: Name
at Raven.Database.DocumentDatabase.AssertPutOperationNotVetoed(String key, RavenJObject metadata, RavenJObject document, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 938
at Raven.Database.DocumentDatabase.<>c__DisplayClass4b.b__43(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 780
at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 608
at Raven.Database.DocumentDatabase.Put(String key, Etag etag, RavenJObject document, RavenJObject metadata, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 843
at Raven.Database.Extensions.CommandExtensions.Execute(ICommandData self, DocumentDatabase database, BatchResult batchResult) in c:\Builds\RavenDB-Stable\Raven.Database\Extensions\CommandExtensions.cs:line 54
at Raven.Database.DocumentDatabase.ProcessBatch(IList`1 commands) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2086
at Raven.Database.DocumentDatabase.<>c__DisplayClass10c.b__108(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2048
at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 666
at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Stable\Raven.Database\Storage\Esent\TransactionalStorage.cs:line 617
at Raven.Database.DocumentDatabase.Batch(IList`1 commands) in c:\Builds\RavenDB-Stable\Raven.Database\DocumentDatabase.cs:line 2051
at Raven.Client.Embedded.EmbeddedDatabaseCommands.Batch(IEnumerable`1 commandDatas) in c:\Builds\RavenDB-Stable\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 709
at Raven.Client.Document.DocumentSession.SaveChanges() in c:\Builds\RavenDB-Stable\Raven.Client.Lightweight\Document\DocumentSession.cs:line 664
at Octopus.Server.Legacy.LegacyOctopusImporter.<>c__DisplayClass18d`1.b__18c(List`1 batch) in y:\work\refs\heads\master\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 916
--- End of inner exception stack trace ---
at Octopus.Server.Legacy.LegacyOctopusImporter.<>c__DisplayClass18d`1.b__18c(List`1 batch) in y:\work\refs\heads\master\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 921
at Octopus.Server.Legacy.LegacyOctopusImporter.LoadAll(String prefix, Action`1 paginator) in y:\work\refs\heads\master\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 990
at Octopus.Server.Legacy.LegacyOctopusImporter.DoTheMigration(Nullable`1 tentaclePort, String upnSuffix) in y:\work\refs\heads\master\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 218
at Octopus.Server.Legacy.LegacyOctopusImporter.ImportFromBackup(String ravenDbBackupFilePath, Nullable`1 tentaclePort, String upnSuffix) in y:\work\refs\heads\master\source\Octopus.Server\Legacy\LegacyOctopusImporter.cs:line 91
at Octopus.Server.Commands.MigrateCommand.Start() in y:\work\refs\heads\master\source\Octopus.Server\Commands\MigrateCommand.cs:line 116
at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in y:\work\refs\heads\master\source\Octopus.Shared\Startup\AbstractCommand.cs:line 55
at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown) in y:\work\refs\heads\master\source\Octopus.Shared\Startup\ConsoleHost.cs:line 36

From the RavenDB.txt log file

2014-06-17 08:18:23.0019 WARN Failed to index UserByTeamIndex
2014-06-17 08:18:23.0019 WARN Failed to index UsersByLoginIndex
2014-06-17 08:18:23.0258 WARN Failed to index TeamsIndex
2014-06-17 08:18:23.0378 WARN Failed to index ReferencedDocumentsIndex
2014-06-17 08:18:44.1489 WARN Could not open index ApiKeysByUserIndex. Recovery operation failed, forcibly resetting index
System.InvalidOperationException: Index does not exists: ApiKeysByUserIndex
at Raven.Database.Indexing.IndexStorage.OpenOrCreateLuceneDirectory(IndexDefinition indexDefinition, String indexName, Boolean createIfMissing) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\IndexStorage.cs:line 315
at Raven.Database.Indexing.IndexStorage.OpenIndexOnStartup(String indexName) in c:\Builds\RavenDB-Stable\Raven.Database\Indexing\IndexStorage.cs:line 126

etc etc etc

My most recent attempt was Octopus Deploy 1.6.3.1723 To Version 2.5.5.318.

Every single restore attempt rant into this error. Now I have see articles that say delete the duplicate data without telling you how to find the duplicate data.

After messing about with various things I stumbled onto a reasonably simple solution and while it is not perfect and possibly not even recomended it solved my restore issues.

This is what I did
My Octopus directory is set as E:\OctopusStore\

A few times I had to remove my activity logs (Backed up the folder just in case) because of

Creating default Octopus Server certificate
Found existing activity log files in 'E:\OctopusStore\OctopusServer\ActivityLogs', these will be renamed to '*.old' files so that they don't match new task IDs.
-------------------------------------------------------------------------------
Error: Cannot create a file when that file already exists.

-------------------------------------------------------------------------------

Activity logs are located here E:\OctopusStore\OctopusServer\ActivityLogs on my install

The raven DB is located in E:\OctopusStore\RavenDB
I deleted the E:\OctopusStore\RavenDB folder and then ran the “Restore from octopus 1.6…” using one of my backups from Octopus 1.6.x.

The result was a working copy of octopus with a few draw backs.
All users were effectively guest users and although I could login I could not see any projects or admin the system.

Along comes RavenDB access.
You can get the web interface up by clicking the “Browse RavenDB” link on the Octopus Manager or simply by navigating your silverlight enabled browser to http://localhost:10931/raven/studio.html.

Once you have that running you just need to find your user id, which is stored in the users collection. Note the Id is in the format “users-x” with x being an integer e.g. users-1.

Now that you know your user Id you just need to update the “Teams” collection with your user details.

You can proceed to making yourself an admin again.

In the Teams collection, open the teams-administrators document.

The newly restored document looks like this (for me)

{
"Name": "Octopus Administrators",
"MemberUserIds": [],
"ExternalSecurityGroups": [],
"UserRoleIds": [
"userroles-systemadministrator"
],
"ProjectIds": [],
"EnvironmentIds": []
}

To make myself an admin was simply a matter of adding my user (“users-1”) to the “MemberUserIds” array. The updated document, for me looks like this

{
"Name": "Octopus Administrators",
"MemberUserIds": [
"users-1"
],
"ExternalSecurityGroups": [],
"UserRoleIds": [
"userroles-systemadministrator"
],
"ProjectIds": [],
"EnvironmentIds": []
}

Now after a refresh of the web interface, located at http://localhost/app#/ I have full control with all my history, environments etc.

Of course in my situation I need to update the tentacles on all the environments AND the Team City plugin as it was updated for the Octopus 2.x release.

Sooo AppFabric

AppFabric simple check cs file
So I was playing with appfabric and ran into some interesting bits.

I discovered that if you have appfabric running on multiple non clustered machines and using MS SQL as the configuration store then.
If you shut down the SQL server
* The distributed cluster still functions
* You can not use the powershell interface to interact with any of the distributed nodes.
* You can however interact with each member via code, in my case C#

Now to switch config servers is relatively easy.

My scenario.
Have 4 nodes running in non clustered mode, I did not have win datacenter or enterprise installed and was not looking for the hig availability at the server level.
No config server.

Result wanted.
The ability to restart the nodes as needed, which meant a new config server being put in place.

Solution.
Take one node offline.
Using the appfabreic config tool, create and setup a new config database in SQL. To Microsofts credit this is really simple to do, just use the appfabric configuration tool (Configure AppFabric is the name in the start menu)

Once I had completed this step it was a matter of re-configuring the other nodes. Since the system was now under load I had to project expected downtime. Which was reasonable easy based on the ease or reconfiguration. 5 minutes downtime expected.

Having a look over the MS powershell configuration guide I checked to see how I can verify all is working as expected.

Verification, well a simple Get-CacheStatistics -CacheName NAME_OF_CACHE is as easy as it gets.

So first node gets started up again and we create the named cached we need using New-Cache -CacheName NAME_OF_CACHE

Right we all set to roll over the other nodes to the new config server.

Going through the “Configure AppFabric” tool and proceeding right to the end but not clicking the final button to do the actual configuration. This way all nodes are ready to roll over. Quick test on a VM shows the service will be restarted and reconfigured. Ok then we wait till a quiet time.

Then it was time, buttons pressed and I could see the machines chugging away as they switch to the new config server. Then bam alarms start to trigger, something is wrong. A quick investigation showed that one of the named cached was not doing as expected so a quick Remove-Cache -CacheName NAME_OF_CACHE then recreate it New-Cache -CacheName NAME_OF_CACHE and tada alarms go quiet.

A quick check that each node can infact use the powerchell inteface.

Get-CacheStatistics -CacheName NAME_OF_CACHE

All seemed to be good at that point. So a super quick console app in C# to test and we were done.

You will of course need the nuget package ServerAppFabric.Client to make this work.

The console code looks like this

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.ApplicationServer.Caching;

namespace AppfabricTest
{
internal class Program
{
public static List cacheNameList = new List();

private static void Main(string[] args)
{
// Names of named caches
cacheNameList.Add(“TEST”);

GetCacheInfo();
}

private static void GetCacheInfo()
{
StreamWriter file = new System.IO.StreamWriter(“Details.txt”, true);

try
{
// Declare array for cache host(s) – in our case just one to quickly test
DataCacheServerEndpoint[] servers = new DataCacheServerEndpoint[1];
servers[0] = new DataCacheServerEndpoint(“localhost”, 22233);

// Setup the DataCacheFactory configuration.
DataCacheFactoryConfiguration factoryConfig = new DataCacheFactoryConfiguration();
factoryConfig.Servers = servers;

factoryConfig.SecurityProperties = new DataCacheSecurity(
DataCacheSecurityMode.None,
DataCacheProtectionLevel.None);

// Create a configured DataCacheFactory object.
DataCacheFactory mycacheFactory = new DataCacheFactory(factoryConfig);

try
{
foreach (var cacheName in cacheNameList)
{
WriteInfo(string.Format(“Using: {0}”, cacheName), file);

// Get a cache client for the default cache
//DataCache myCache = mycacheFactory.GetDefaultCache(); //or change to mycacheFactory.GetCache(myNamedCache);
DataCache myCache = null;

try
{
myCache = mycacheFactory.GetCache(cacheName);

WriteInfo(string.Format(“Inserting dummy data into: {0}”, cacheName), file);

//inserty dummytest data to test
myCache.Put(“key1”, “test1”);
myCache.Put(“key2”, “test 2”);
myCache.Put(“key3”, “test three”);

//read back our values
string result1 = (string)myCache.Get(“key1”);
string result2 = (string)myCache.Get(“key2”);
string result3 = (string)myCache.Get(“key3”);

WriteInfo(string.Format(“Retrieved result1: {0}”, result1), file);
WriteInfo(string.Format(“Retrieved result2: {0}”, result2), file);
WriteInfo(string.Format(“Retrieved result3: {0}”, result3), file);

//list all items in the cache : important part
foreach (string region in myCache.GetSystemRegions())
{
WriteInfo(string.Format(“Cache name: {0} — Region name: {1}”, cacheName, region), file);

try
{
foreach (var kvp in myCache.GetObjectsInRegion(region))
{
string content =
String.Format(
“Cache: {0} — data item (‘{1}’,'{2}’) in region {3} of cache {4}”,
cacheName,
kvp.Key,
kvp.Value.ToString(),
region,
cacheName);

WriteInfo(content, file);
}
}
catch (Exception exItem)
{
WriteInfo(string.Format(“Error 1: {0} – {1}”, cacheName, exItem.Message), file);
}
}
}
catch (Exception)
{
WriteInfo(string.Format(“Failed to GetCache: {0}”, cacheName), file);
myCache = null;
}
}
}
catch (Exception exItem)
{
WriteInfo(string.Format(“Error 2: {0}”, exItem.Message), file);
}

}
catch (Exception exItem)
{
WriteInfo(string.Format(“Error 3: {0}”, exItem.Message), file);
}
finally
{
file.Close();
file.Dispose();
}
}

static void WriteInfo(string message, StreamWriter file)
{
Console.WriteLine(message);
file.WriteLine(message);
}
}
}

AppFabric simple check cs file

How to move mysql data directory on ubuntu 14.04

Moving the data dir for mysql in ubuntu 14.04

#Shutdown mysql
sudo service mysql stop

#change the data directory
sudo nano /etc/mysql/my.cnf

#Look for the “datadir” setting and set your new path
#Save the file after the changes are made

#update apparmor
nano /etc/apparmor.d/usr.sbin.mysqld

#Add to the end of the file
# NOTE: without the trailing slash “/” on /db/mysql/ you will get the dreaded error 13 permission denied error
# Also note that the path listed below was my path and sould be set to wherever you move your data directory to.

/db/mysql/ r,
/db/mysql/** rwk,

#Restart apparmor
sudo service apparmor restart

#then start up mysql
sudo service mysql start

#That should do it

#A quick test to verify it worked …
mysql -uYOUR_USERNAME -pYOUR_PASSWORD -e ‘show databases;’

#or if you dont want to put your password in a remembered command line
mysql -uYOUR_USERNAME -p -e ‘show databases;’