Jump to content

Virtual copilot addon


Peter Pukhnoy

Recommended Posts

Luke Hall

Hi @Peter Pukhnoy

I'm trying to use the Copilot addon with voice control for the first time but there seems to be no communication between my windows speech recognition and the sim.

Copilot isn't hearing anything I say to activate checklists or anything else. SR is working in Windows normally and my microphone is correctly setup.

VC is enabled in copilot but there aren't even any references to voice commands in the log file.

I'm sure that I've fundamentally misunderstood something that I'm supposed to do here! What though...?

Link to comment
Peter Pukhnoy
8 minutes ago, Luke Hall said:

VC is enabled in copilot but there aren't even any references to voice commands in the log file.

Can you attach the log please?

Link to comment
Luke Hall
2 minutes ago, Peter Pukhnoy said:

Can you attach the log please?

Sure, thanks. The last bit of the log says speech disabled because I'm doing a flight now with it turned off! :-)

Copilot.log

Link to comment
Peter Pukhnoy
4 hours ago, Luke Hall said:

Sure, thanks. The last bit of the log says speech disabled because I'm doing a flight now with it turned off! :-)

[18:40:50] [error] Error changing rule state: 0x88890008

I've already had someone message me about the same error. The culprit turned out to be Sonic Studio, the same software that you're using. Maybe it's possible to initialize the recognizer differently in a way that would be compatible with SS, but I don't know how (if someone does, let me know). Unfortunately, the only way to resolve this is to turn off/uninstall Sonic Studio.

Even FS2Crew seems to behave the same with Sonic Studio.

Link to comment
  • 2 weeks later...
Luke Hall
On 5/26/2021 at 8:07 PM, Peter Pukhnoy said:

[18:40:50] [error] Error changing rule state: 0x88890008

I've already had someone message me about the same error. The culprit turned out to be Sonic Studio, the same software that you're using. Maybe it's possible to initialize the recognizer differently in a way that would be compatible with SS, but I don't know how (if someone does, let me know). Unfortunately, the only way to resolve this is to turn off/uninstall Sonic Studio.

Even FS2Crew seems to behave the same with Sonic Studio.

Thanks @Peter Pukhnoy. I have removed Sonic Studio (I never liked it anyway) and all is well again!

Link to comment
  • 2 weeks later...
JohnHowe

Hi All,

Anyone having problems with Copilot in 5.2 and the latest update from FsLabs?

I get the following errors in ContentErrors.txt:-

[error.0]
error=Error loading Copilot.dll: The specified procedure could not be found.


[error.1]
error=Error loading "Copilot for FSLabs" from "C:\Users\xxx\Documents\Prepar3D v5 Add-ons\Copilot for FSLabs\Copilot.dll".


 

Link to comment
Peter Pukhnoy
3 hours ago, JohnHowe said:

Anyone having problems with Copilot in 5.2 and the latest update from FsLabs?

No problems here B)

Which versions of Copilot? Have you tried reinstalling it?

Link to comment
JohnHowe

Hi Peter,

It is 2.2.1.0, the very same version I was running with P3D 5.1 and the previous build of FSLabs.  When I re-installed P3D I did not delete the Prepar3D v5 Add-ons folder, left it exactly as it was.

Deleted my previous installation, ran P3D, the copied over the contents of the ZIP ran P3D, enabled the add-on and still get the same errors.

 

Link to comment
Peter Pukhnoy
19 hours ago, JohnHowe said:

Hi Peter,

It is 2.2.1.0, the very same version I was running with P3D 5.1 and the previous build of FSLabs.  When I re-installed P3D I did not delete the Prepar3D v5 Add-ons folder, left it exactly as it was.

Deleted my previous installation, ran P3D, the copied over the contents of the ZIP ran P3D, enabled the add-on and still get the same errors.

Are you sure that it was the P3D update that caused the problem? I struggle to imagine how that's possible. 

My primary suspect would be an anti-virus program. If you have one installed, try temporarily disabling it.

It might be worth trying to reinstall the vc++ redistributables (vc_redist.x64.exe)

Link to comment
JohnHowe

Hi Peter,

Tried that, no joy I'm afraid.  Have removed the add-on and will try again when the next hot-fix from LM comes out.

J

 

Link to comment
Peter Pukhnoy
On 6/17/2021 at 4:55 PM, JohnHowe said:

Hi Peter,

Tried that, no joy I'm afraid.  Have removed the add-on and will try again when the next hot-fix from LM comes out.

J

 

Can try again with ActiveSky disabled?

I've been able to replicate the same error after installing AS and will provide a fix in the next couple of days.

Link to comment
JohnHowe

Hi Peter,

That might be it.  I did install the latest version.  Is this just installed or AS running when P3d runs up?

 

 

Link to comment
Peter Pukhnoy
2 hours ago, JohnHowe said:

Is this just installed or AS running when P3d runs up?

The AS connector add-on running.

Link to comment
David OConnor

Hey Peter,

I'm gonna get right to the point, I love the virtual copilot for FSLabs A320 family, still getting used to it. I am having a slight issue with the checklist commands via voice control, after completing "Before Takeoff to the line checklist" I am unable to start "Before Takeoff below the line checklist" even though I completed every checklist from before start to before takeoff to the line, can you help me with this bug or confusion?

I also like to add that I am unable to open the in sim menu via a key binding.

P.S. I don't have alot of knowledge about key binding or coding.

Thanks.

Link to comment
Peter Pukhnoy
2 hours ago, Mardoc1994 said:

I am unable to start "Before Takeoff below the line checklist" even though I completed every checklist from before start to before takeoff to the line, can you help me with this bug or confusion?

Are you letting the PM do the lineup flow before requesting below the line?

Link to comment
John Rooum

Peter
I am now seeing this same problem.  All checklists etc seem ok except the "Before Take Off Below the Line". I also found that the Line Up Procedure flow does not work using either voice or manual trigger. But the Take Off and all subsequent procedures and checklists or OK.

Using P3Dv5.1,  FSLabs 5.0.1.160 and copilot versions 2.2.2 and then reverted to 2.1.1 Both with the same issue

Latest log attached - problem is around 1222.Grateful if you could see anything I am missing.

Many Thanks

 

Copilot.log

Link to comment
Peter Pukhnoy
26 minutes ago, John Rooum said:

I also found that the Line Up Procedure flow does not work using either voice or manual trigger

Sorry for the stupid question, but can you confirm you're doing the manual trigger correctly? For example, when the seatbelts sign switch is ON, the trigger will be moving the switch OFF->ON->OFF->ON within 2 seconds.

Link to comment
John Rooum

I might have got that wrong as I am using voice. I will run it again in an hour - one question `- when should we do the below the line check - before or after the line up procedure?

Link to comment
Peter Pukhnoy
5 minutes ago, John Rooum said:

before or after the line up procedure?

After. The recognizer starts listening for "below the line" after the lineup procedure.

7 minutes ago, John Rooum said:

I might have got that wrong as I am using voice

The manual trigger is the same with or without voice. The log indicates that you were using the manual trigger.

Link to comment
John Rooum

Thanks Peter

"After. The recognizer starts listening for "below the line" after the lineup procedure"

Thought that might be the case looking at the items in Below the Line.

Setting up now

Link to comment
John Rooum

Yep - no problems when I completed line up procedures first - other than the vouch trigger was not recognised and I had to use manual trigger.

anyway time to update to P3D 5.2 and no doubt a whole lots of new problems

thanks again

Link to comment
Peter Pukhnoy
6 minutes ago, John Rooum said:

other than the vouch trigger was not recognised and I had to use manual trigger.

You can try lowering the confidence threshold (default is 0.94 for the lineup command):

copilot.voiceCommands.lineup:setConfidence(0.85)

Paste the code into a lua file in copilot\custom.

Link to comment
  • 1 month later...
Stirling Day

Hi Peter, I have uplifted extra fuel that wasn't on the loadsheet, and it seems like the F/O is getting stuck here. I requested 15.0 on the OFP data page, but upped it to 15.6 using the fuel panel once we had started fuelling. GSX fuelled to 15.6 and now we are stuck here on the before start.

 

Thanks again,

SD

EXTRA UPLIFT.png

Link to comment
Peter Pukhnoy
15 minutes ago, Stirling Day said:

Hi Peter, I have uplifted extra fuel that wasn't on the loadsheet, and it seems like the F/O is getting stuck here. I requested 15.0 on the OFP data page, but upped it to 15.6 using the fuel panel once we had started fuelling. GSX fuelled to 15.6 and now we are stuck here on the before start.

 

Thanks again,

SD

 

I don't think there's any way to handle this other than to skip the checklist item, which you can do through the checklist menu (assign a key with menu_keybind in options.ini).

  • Like 1
Link to comment
Stirling Day
Just now, Peter Pukhnoy said:

I don't think there's any way to handle this other than to skip the checklist item, which you can do through the checklist menu (assign a key with menu_keybind in options.ini).

Okay, that works, thank you so much for the fast response! 

Have a good one.

Link to comment
Aidan le Gras

Hi Peter.

Fantastic add-on, thanks very much for going to the trouble of setting it up. I am having some trouble though as I get used to the peculiarities. With reference to the attached log, forgive the barrage but could you answer me the following questions please?

1. Why does it not recognise my request to commence the "before takeoff checklist"?

2. Why does it not recognise my "line-up" request i.e. why will it not select the transponder to TA/RA and landing lights to on.

3. Why does it not recognise my call for "after takeoff climb-checklist" or "landing checklist"?

4. Is the application set-up to:

recognise the call for "go-around flaps",

respond with 1 stage less,

call new flap setting and then await an FMA callout for the subsequent 'positive climb' callout?

5. If some of the issues I am having are simply of changing the confidence level to 0.8 as you recommend in many other instances, could I not simply do that in the main files rather than create a custom file?

And lastly, are you planning to make this a pay-ware app? It will, in time, be well worth it in my view.

Copilot.log

Link to comment
Peter Pukhnoy
5 hours ago, Aidan le Gras said:

1. Why does it not recognise my request to commence the "before takeoff checklist"?

2. Why does it not recognise my "line-up" request i.e. why will it not select the transponder to TA/RA and landing lights to on.

Because you never completed the after start checklist. You said 'standby checklist' and never said 'resume checklist'.

5 hours ago, Aidan le Gras said:

3. Why does it not recognise my call for "after takeoff climb-checklist"

It did, but I made it so you can't do two checklists at the same time and you still had the after start checklist on standby :)

5 hours ago, Aidan le Gras said:

4. Is the application set-up to:

recognise the call for "go-around flaps",

respond with 1 stage less,

call new flap setting and then await an FMA callout for the subsequent 'positive climb' callout?

Currently all it does is retract the flaps 1 step, but I'll add the other items.

5 hours ago, Aidan le Gras said:

5. If some of the issues I am having are simply of changing the confidence level to 0.8 as you recommend in many other instances, could I not simply do that in the main files rather than create a custom file?

Yes, there is a global confidence option in options.ini, but I don't think you need that yet.

5 hours ago, Aidan le Gras said:

And lastly, are you planning to make this a pay-ware app? It will, in time, be well worth it in my view.

No, but I'll consider adding a donation link.

  • Thanks 1
Link to comment
Aidan le Gras

@Peter Pukhnoy fantastic, thanks so much for the detailed response, sounds like I’m not miles away from getting it all together, I’ll see if I can have go and change the response to V1,VR,V2 and flex temp settings to ‘Set’ as per our SOP’s.

On a side note, another group entity that also operates Airbus responds in the way that you propose (with each value) but not us. We also don’t respond with takeoff weight C of G% on the after start checklist but rather units of pitch trim up or down. Again, there’s a lot to be said for the % method because of incidents such a this: 

https://www.aerohabitat.org/airmanshiponline/23-4-2002/pdf/19-NTSB-A3xx center-of-gravity.pdf

TL/DR summary, if you set THS units down instead of up as usually occurs on the 320 but less so on the 321 you may end up having some difficulty during rotation. That can’t happen if there’s no likelihood of mixing up the ve+/ve- value of the THS setting. Although, later FMGC standards will detect gross error miss settings. 
 

Some other observations if I may, it appears that the PM is waiting until the gear movement is completed which is not necessary before stating ‘gear up’/ ‘gear down’:

‘The PM selects the L/G lever position and replies
after checking the red lights on the LDG GEAR
indicator to confirm gear operation.’

 

I was also wondering if there is an ‘activate approach phase’ request that is actioned by the FO?

Thanks again Peter, great job!

Link to comment
Peter Pukhnoy
18 hours ago, Aidan le Gras said:

I’ll see if I can have go and change the response to V1,VR,V2 and flex temp settings to ‘Set’ as per our SOP’s.

That would be pretty easy:

copilot.checklists.beforeTakeoff:replaceItem("takeoffSpeedsFlexTemp", {
    response = VoiceCommand:new("set")
})

Although this doesn't include any validation. Do you want any?

18 hours ago, Aidan le Gras said:

We also don’t respond with takeoff weight C of G% on the after start checklist but rather units of pitch trim up or down

I'll consider adding it as an option in the future

18 hours ago, Aidan le Gras said:

Some other observations if I may, it appears that the PM is waiting until the gear movement is completed which is not necessary before stating ‘gear up’/ ‘gear down’:

I don't understand what you're referring to here. The PM doesn't say these callouts at all in my addon.

18 hours ago, Aidan le Gras said:

I was also wondering if there is an ‘activate approach phase’ request that is actioned by the FO?

No, but it's something you can program:

VoiceCommand:new({
  phrase = "activate approach phase",
  action = function(vc)
    FSL.PED_MCDU_KEY_PERF()
    copilot.sleep(500, 1500)
    while FSL.MCDU:getString():find("APPR PHASE") do
      FSL.PED_MCDU_LSK_L6()
      copilot.sleep(500, 1500)
    end
  end
}):activateOn(copilot.events.belowTenThousand)

 

Link to comment
Aidan le Gras
7 hours ago, Peter Pukhnoy said:

That would be pretty easy:

copilot.checklists.beforeTakeoff:replaceItem("takeoffSpeedsFlexTemp", {
    response = VoiceCommand:new("set")
})

Although this doesn't include any validation. Do you want any?

I'll consider adding it as an option in the future

I don't understand what you're referring to here. The PM doesn't say these callouts at all in my addon.

No, but it's something you can program:

VoiceCommand:new({
  phrase = "activate approach phase",
  action = function(vc)
    FSL.PED_MCDU_KEY_PERF()
    copilot.sleep(500, 1500)
    while FSL.MCDU:getString():find("APPR PHASE") do
      FSL.PED_MCDU_LSK_L6()
      copilot.sleep(500, 1500)
    end
  end
}):activateOn(copilot.events.belowTenThousand)

 

That’s awesome, thanks for the response. 
 

1st point:

Useful validation would ultimately be great in time: checking that speeds are in MCDU (without ‘confirm takeoff speeds’ option at LSK 6R) and either speeds are displayed on PFD or FCU speed window NOT showing 100, as often happens after an FD being selected off/FMGC reset) I’ll throw those lines in there as you stated for now. That’s great, thanks. 
 

2nd point: no problem at all. We’re required to confirm that the THS and pitch trim match when we check the load data in any case but at the point of checklist confirmation we’re matching MCDU THS figures to the same data point on the trim wheel. (units up or down). 
 

Third point: 

21 hours ago, Aidan le Gras said:

The PM selects the L/G lever position and replies
after checking the red lights on the LDG GEAR
indicator to confirm gear operation.’


So when I didn’t hear PM callout ‘gear up’ after my request for ‘gear up’ I was assuming he was going to do it after it was fully retracted, I can see now that he doesn’t do it at all. IRL it is a standard call for the PM to respond with both ‘gear up’ and ‘gear down’ in response to the PF request and the subsequent red lights on the LG panel:

‘The PM selects the L/G lever position and replies
after checking the red lights on the LDG GEAR
indicator to confirm gear operation.’
 (This applies for up and down)

In the event of an engine failure after takeoff with a FLX temp set, the PM calling ‘gear up’ happens to be the perfect time to select MAN TOGA as it compensates almost perfectly, at least in the sim, with the extra drag from the gear doors opening that appears at exactly the time the gear lights go red for retraction, hence minimising SRS fluctuations which will get you greater risk of ‘DON’T SINK’ callout and potentially a requirement to repeat the exercise. 

As for the last point, that’s great thanks, I’ll use that too. 
 

The QRH flow shows that the PF does it but in the interests of not going ‘heads down’ most simply call for it or let it self activate in the interests of not going ‘heads down’.

I’m going to have to start trying to learn some of this LUA scripting I think, it’s all Greek to me. Thanks again. 
 

 

Link to comment
Peter Pukhnoy
9 hours ago, Aidan le Gras said:

either speeds are displayed on PFD or FCU speed window NOT showing 100

It's not possible to read the displays (other than the MCDU), but you can do the FCU check:

copilot.checklists.beforeTakeoff:replaceItem("takeoffSpeedsFlexTemp", {
    response = VoiceCommand:new("set"),
    onResponse = function(check)
      check(not FSL.FCU:get().SPD, "speed isn't dashed")
    end
})

 

Link to comment
Aidan le Gras
6 hours ago, Peter Pukhnoy said:

It's not possible to read the displays (other than the MCDU), but you can do the FCU check:

copilot.checklists.beforeTakeoff:replaceItem("takeoffSpeedsFlexTemp", {
    response = VoiceCommand:new("set"),
    onResponse = function(check)
      check(not FSL.FCU:get().SPD, "speed isn't dashed")
    end
})

 

Understood, you’ve done a great job with what you’ve got Peter, thanks for your help. 

Link to comment
  • 3 months later...
David OConnor

Hi Peter,

I have reported in the past about small hiccups with your co-pilot module and I am getting the hang of it. But I have mentioned before about opening the menu in the sim and I have bind a key or two so I could gain access to the menu but while in the sim when I press the assigned key(s) the menu does not open or appear on screen.

I use your feature alot and I have to say it is quite impressive and useful, great job keep it up.

Thanks,

Mardoc.

Link to comment
Peter Pukhnoy
50 minutes ago, Mardoc1994 said:

But I have mentioned before about opening the menu in the sim and I have bind a key or two so I could gain access to the menu but while in the sim when I press the assigned key(s) the menu does not open or appear on screen.

I have just uploaded a new version which fixes an issue with the menu. However the issue wasn't the menu not opening at all. Nevertheless, please try the new version and report back. Do you have other addons that use the SimConnect menu (for example, GSX or UGCX) to confirm that it works for you in general?

Link to comment
David OConnor

Thanks for your reply, I have downloaded the new version and I have discovered during "Flight Control Checks" that the copilot won't say anymore control position after the first input of control e.g. when I input full left aileron the copilot will say "Full Left" as normal, then when I input full right aileron the copilot does not say "Full Right" at all, same goes for elevator inputs.

Link to comment
Peter Pukhnoy
1 hour ago, David OConnor said:

Thanks for your reply, I have downloaded the new version and I have discovered during "Flight Control Checks" that the copilot won't say anymore control position after the first input of control e.g. when I input full left aileron the copilot will say "Full Left" as normal, then when I input full right aileron the copilot does not say "Full Right" at all, same goes for elevator inputs.

Try now please Copilot.zip

Link to comment
×
×
  • Create New...