Thursday, December 5, 2013

Task 22. SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER


This week I had another SSIS bug to kill. :)

I have 3 SSIS packages: 2 child and 1 master


The Control Flow for the first one is integrated in a Sequence Container to having the rollback ability. 

When editing the Sequence Container properties there is TransactionOption set to Required. 

For the second package we have another sequence container with the same options.
But the Master one has 2 Execute Package Task only.

In the Properties window of Master package there is the same option named TransactionOption set to Supported.


So, all these packages are working fine on the server.
The problem appears when moving them to another server: the client server

After modifying all the connection strings for all the packages an error appears:

[OLE DB State Source [1565]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.  The AcquireConnection method call to the connection manager "DDBillingPortal_TempLive.dd" failed with error code 0xC0202009.  There may be error messages posted before this with more information on why the AcquireConnection method call failed.

I was searching through a lot of articles but nobody was offering a solution to this error. Finally, the problem is related to that TransactionOption set to Required instead of Supported in those 2 packages.



Tuesday, August 13, 2013

Microsoft Certifications



 I just had a discussion with someone regarding the value of Microsoft certifications and what exactly does it take to become MCSA certified?  Last year Microsoft launched their new MCSA -- Microsoft Certified Solutions Associate certification. While entry-level may be able to successfully sit for the first exam, by the time you get to the second or third, you'll likely need at least 2 years of experience to pass. The MCSA contains 3 certifications you need to pass: 70-461, 70-462 and 70-463, but there are more to stand for like 70-464 and 70-465. Who knows maybe until next year the smart folks from Microsoft will release another 5 certifications :-) 


Exam 70-464:

Developing Microsoft SQL Server 2012 Databases

http://www.microsoft.com/learning/en/us/exam.aspx?id=70-464

Exam 70-465:

Designing Database Solutions for Microsoft SQL Server 2012

https://www.microsoft.com/learning/en/us/exam.aspx?ID=70-465

Tuesday, April 2, 2013

Task 21. Data Processing Techniques





Ever wondered how to process millions of rows daily?
SQL Server 2012 comes with new TSQL functions and index enhancement to help solve high volume data processing with techniques that will be kind to the server. 

The main problem is related to:
 1. High Volume Data ( size is relative)
2. Business Requirements ( complex business logic, duration, data availability, scalability, configurable, rerunable)

The DBA becomes crazy when comes about disk space, memory usage and CPU bottlenecks. They need to care about availability/response time, risks reduction/avoidance and governance.

You need to know all the details about SQL Server environment: information about recovery model, utilisation, hardware environment: number of processors, memory, network speed, disk configuration.

Useful techniques:
1. Reliable pipe ( batch processing using T-SQL)
2. Fast pipe (SSIS, BCP, BULK INSERT)
3.Divide and conquer (Balanced data distributor with SSIS, Partitioning)

The idea of batching in T-SQL is to break the problem into smaller pieces:
1. Table Lock duration reduced
2. Less rows to rollback
3.Less TempDB consumption
4.Control on error handling
5. Customised visibility on progress


Batching SQL is great for:
1. Transferring data in the same instance
2. Low priority data transfer with long duration
3.Batching scripts can be converted to stored procedures - ideal for replication
4. TempDB size is limited
5. Transaction Log is generally small for SIMPLE or BULK LOGGED recovery mode

Break large delete operations into chunks  is an article for daily operations data process.

There are some tricky scenarios that can not be done using batching like get a unique list of number of columns from 100 million rows or joining 100 million rows with another 100 million rows on nvarchar(500) columns.

In SQL 2012 there are column store indexes which group store data for each column and join all the columns to complete the whole index. Enable faster performance for common data warehousing queries such as filtering, aggregating, grouping and star-join queries.

CREATE NONCLUSTERED COLUMNSTORE INDEX [ISCS_DailySpeed_ColumnStore] ON [dbo].[DailySpeed_ColumnStore] ( [LocationID], [CaptureDateID],[DownloadKbps], [UploadKbps]) WITH (DROP_EXISTING = OFF) ON [PRIMARY]
GO

SSIS -Data flow task
Minimally Logged Inserts
1. Transaction Log keeps track of extent allocations and metadata changes only




Friday, December 7, 2012

Task 20: NoSQL vs SQL


SQL vs NoSQL, they’re actually talking about relational versus non-relational databases. That’s what I’m talking about here: Relational databases (such as Oracle, MySQL, and SQL Server) versus newer non-relational databases (such as MongoDB, CouchDB, BigTable, and others).



Popular open source NoSQL data stores include:
  • Cassandra (tabular, written in Java, used by Cisco WebEx, Digg, Facebook, IBM, Mahalo, Rackspace, Reddit and Twitter)
  • CouchDB (document, written in Erlang, used by BBC and Engine Yard)
  • Dynomite (key-value, written in Erlang, used by Powerset)
  • HBase (key-value, written in Java, used by Bing)
  • Hypertable (tabular, written in C++, used by Baidu)
  • Kai (key-value, written in Erlang)
  • MemcacheDB (key-value, written in C, used by Reddit)
  • MongoDB (document, written in C++, used by Electronic Arts, Github, NY Times and Sourceforge)
  • Neo4j (graph, written in Java, used by some Swedish universities)
  • Project Voldemort (key-value, written in Java, used by LinkedIn)
  • Redis (key-value, written in C, used by Craigslist, Engine Yard and Github)
  • Riak (key-value, written in Erlang, used by Comcast and Mochi Media)
  • Ringo (key-value, written in Erlang, used by Nokia)
  • Scalaris (key-value, written in Erlang, used by OnScale)
  • Terrastore (document, written in Java)
  • ThruDB (document, written in C++, used by JunkDepot.com)
  • Tokyo Cabinet/Tokyo Tyrant (key-value, written in C, used by Mixi.jp (Japanese social networking site))


db4o (database for objects) is an embeddable open source object database for Java and .NET developers. It is developed, commercially licensed and supported by Versant.
db4o is written in Java and .NET and provides the respective APIs. db4o can run on any operating system that supports Java or .NET. 




Thursday, November 29, 2012

Task 19: Update/Insert procedure

                                                            

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[spUpdateInsert]
AS
SET NOCOUNT ON; 
BEGIN TRY
BEGIN TRANSACTION
DECLARE @ERROR_MESSAGE NVARCHAR(100) = ERROR_MESSAGE();
DECLARE @ERROR_SEVERITY INT = ERROR_SEVERITY();


UPDATE ZIP
SET 
ZIP.[ZipCode] = ZC.[PostalCode],
ZIP.[CityID] = C.[ID],
ZIP.[StateID] = S.[ID],
ZIP.[County] = ZC.[County]
FROM dbo.ZipCode ZIP
INNER JOIN [New].dbo.tblZipCode ZC ON ZIP.[ZipCode] = ZC.[PostalCode]
LEFT JOIN [QA].dbo.[city] c on c.name = zc.city
LEFT JOIN [QA].dbo.[state] s on s.name = zc.[State]
--------------------------------------------------------------

SET IDENTITY_INSERT dbo.ZipCode ON
INSERT INTO dbo.ZipCode(
[ID],
[ZipCode],
[CityID],
[StateID],
[County]
)
SELECT 
ZC.[PostalCodeID],
ZC.[PostalCode],
C.[ID] AS CityID,
S.[ID] AS StateID,
ZC.[County]
FROM dbo.ZipCode ZIP
RIGHT JOIN [New].dbo.tblZipCode ZC ON ZIP.[ZipCode] = ZC.[PostalCode]
LEFT JOIN [QA].dbo.[city] c on c.name = zc.city
LEFT JOIN [QA].dbo.[state] s on s.name = zc.[State]
WHERE ZIP.[ID] is null

SET IDENTITY_INSERT dbo.ZipCode OFF


--SELECT * FROM [New].dbo.tblZipCode

--SELECT * FROM [QA].dbo.ZipCode

--------------------------------------------------------------

UPDATE ZIP
SET 
ZIP.[CityID] = C.[ID],
ZIP.[StateID] = S.[ID]
 
FROM dbo.ZipCode ZIP
INNER JOIN [DieselDirect_New].dbo.tblZipCode ZC ON ZIP.[ZipCode] = ZC.[PostalCode]
LEFT JOIN [DDBillingPortal_QA].dbo.[city] c on c.name = zc.city
LEFT JOIN [DDBillingPortal_QA].dbo.[state] s on s.name = zc.[State]

COMMIT TRAN -- Transaction Success!

END TRY

BEGIN CATCH
    IF @@TRANCOUNT > 0
        ROLLBACK TRAN --RollBack in case of Error
   RAISERROR(@ERROR_MESSAGE,@ERROR_SEVERITY, 1)
END CATCH



Wednesday, October 24, 2012

Task 18: SQL Profiler





     SQL Profiler is the graphical tool that lets you capture and analyzer SQL Server events in real time. I'm going to drill a bit deeper into this useful tool and show you an interesting book by Brad McGehee. You can download it from here.

  • Chapter 01: Getting Started with Profiler
  • Chapter 02: Working with Traces and Templates
  • Chapter 03: Profiler GUI Tips and Tricks
  • Chapter 04: How to Identify Slow Running Queries
  • Chapter 05: How to Identify and Troubleshoot SQL Server Problems
  • Chapter 06: Using Profiler to Audit Database Activity
  • Chapter 07: Using Profiler with the Database Engine Tuning Advisor
  • Chapter 08: Correlating Profiler with Performance Monitor
  • Chapter 09: How to Capture Profiler Traces Programmatically
  • Chapter 10: Profiler Best Practices
  • Chapter 11: Profiler Events and Data Columns Explained

You should also check SQL Profiler Tips and Tricks and SQL Profiler Best Practices

Keyboard Shortcuts

  • Ctrl+Shift+Delete to clear the current trace window
  • Ctrl+F to open the Find dialog box
  • F3 to find the next match
  • Shift+F3 to find the previous match
  • Ctrl+N to open a new trace
  • F5 to start a replay
  • Shift+F5 to stop a replay
  • F9 to toggle a breakpoint
  • F10 to single-step
  • Ctrl+F10 to run to the cursor

A BETTER SQL SERVER PROFILER ALTERNATIVE is accuprofiler  - http://www.accuprofiler.com/




Tuesday, October 23, 2012

Task 17: SSIS with variable excel connection manager



1. In order to deploy an Excel Source with .xlsx extension you need to install the following provider:
http://www.microsoft.com/en-us/download/details.aspx?id=13255

2. Create a new SSIS project (Integration Services Project1)

3. Create a new package (LoadCostChanges.dtsx)

4. Declare 3 variables: UserID, FilePath and VendorUploadID (the variables will be sent as parameters from a stored procedure to the package)



5. Add from toolbox a DataFlow component (Import the Excel File) and an Execute SQL Task.


6. Because the excel file can have multiple empty rows we'll need to delete them using an Execute SQL Task (but this will be the last step for the implementation)




7.  In Data Flow component choose an Excel Source, a Derived Column and an OLE DB Destination component from Toolbox.


8. Excel Source configuration:


9. Excel Connection Manger configuration


10.  Derived Column configuration


11. OLE DB Destination configuration


12. Excel Connection Manager configuration
Connection String:  "Provider=Microsoft.ACE.OLEDB.12.0;Data Source="+ @[User::FilePath] + ";Extended Properties=\"EXCEL 12.0;HDR=YES\";"


13. The dbo.LoadExcel must be created with all the columns from all the excel files if there are columns which can be mapped in more than one Excel Source. 

CREATE TABLE [dbo].[LoadExcel](
[LoadExcelID] [bigint] IDENTITY(1,1) NOT NULL,
[VendorUploadID] [int] NULL,
        ............................................
        [UserID] [uniqueidentifier] NULL,
 CONSTRAINT [PK_LoadExcel] PRIMARY KEY CLUSTERED 
(
[LoadExcelID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
GO


14. You need to connect to your integration server and deploy all your packages in SQL Server:


15. The Stored procedure which call the packages and sent them parameters : 

USE [TFMVendorPortal]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-----------------------------------------------------------------
--settings before running for the first time xp_cmdshell command
-----------------------------------------------------------------
/*
sp_configure 'Show Advanced Options', 1;
RECONFIGURE;
GO
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO
EXEC sp_configure 'show advanced options', 1 
GO
RECONFIGURE WITH OVERRIDE 
GO
EXEC sp_configure 'xp_cmdshell', 1 
GO
RECONFIGURE WITH OVERRIDE 
GO
EXEC sp_configure 'show advanced options', 0 
GO
*/

ALTER PROCEDURE [dbo].[tfm_RunSSISPackages] 
@VendorUploadId int  ,@FilePath varchar(500), @UserID UNIQUEIDENTIFIER
AS
BEGIN

DECLARE @SQL NVARCHAR(max)

 ----TEST SCRIPT 
--DECLARE @SQL NVARCHAR(MAX)
--declare @FilePath nvarchar(max)
--declare @VendorUploadID INT
--SET @VendorUploadId = 233
--set @FilePath = '\\192.168.1.3\tfm\7\vernon_CostChanges_2012-10-15_17-13-33-718.xlsx'

set @SQL = '
EXEC master..xp_cmdshell ''""C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\DTExec.exe" /DTS "\File System\LoadCostChanges" /SERVER "SFVM05" /set \package.variables[User::VendorUploadID].Value;"'+ cast(@VendorUploadID as nvarchar(max))+ '" /set \package.variables[User::FilePath].Value;"'+ @FilePath + '""'''
exec (@SQL)

END