Q:

PowerShell and WinApi

I was trying this:
$signature = @'
[DllImport("shell32.dll")]
public static extern int ShellExecuteW(
  int    hwnd,
  string lpOperation,
  string lpFile,
  string lpParameters,
  string lpDirectory,
  int     nShowCmd
);
'@

$exec = Add-Type -memberDefinition $signature -name "win" -namespace Win32Functions -passThru

$exec::ShellExecuteW(0, "open", "notepad.exe", 0, 0, 1)

but the notepad does not start. How do I write this down correctly?

A:

The immediate fix is twofold:

[DllImport("shell32.dll")] ->
[DllImport("Shell32.dll", CharSet = CharSet.Unicode)]

Since you explicitly target ShellExecuteW - the Unicode version of the API function - you must declare that fact via the CharSet field in the attribute.[1]
While not strictly necessary, int    hwnd should be IntPtr    hwnd.

$exec::ShellExecuteW(0, "open", "notepad.exe", 0, 0, 1) ->
$exec::ShellExecute(0, "open", "notepad.exe", [NullString]::Value, [NullString]::Value, 1)

Note the use of [NullString]::Value to pass null to the string-typed parameters; by default, PowerShell treats $null in a string context as the empty string rather than null; in this particular case, however, the empty string (and therefore $null) would work too.

To put it all together:
$exec = Add-Type -name "win" -namespace Win32Functions -passThru -memberDefinition @'
[DllImport("shell32.dll", CharSet=CharSet.Unicode)]
public static extern int ShellExecuteW(
  IntPtr hwnd,
  string lpOperation,
  string lpFile,
  string lpParameters,
  string lpDirectory,
  int    nShowCmd
);
'@

$exec::ShellExecuteW(
  0, 
  'open', 
  'notepad.exe', 
  [NullString]::Value, 
  [NullString]::Value, 
  1
)

Taking a step back: The Start-Process cmdlet allows you to do the same without the need for on-demand-compiled P/Invoke declarations:
# Use -WorkingDirectory, if  needed.
Start-Process Notepad.Exe -Verb Open -WindowStyle Normal

[1] Omitting the W suffix (ShellExecute) while not specifying a CharSet value works too in practice, although the docs imply that the ANSI version is then called. However, I don't see that in practice: even when passing arguments with characters outside the ANSI range (e.g., 'file€§ü.txt') to the ANSI version of the function explicitly, they seem to be passed through to Notepad correctly. Conversely, however, passing such a string to a console application never seems to pass them through correctly, even when explicitly targeting the Unicode version.

